graph-crawler 5.3.4__tar.gz → 5.3.6__tar.gz

{graph_crawler-5.3.4 → graph_crawler-5.3.6}/CHANGELOG.md

@@ -1,5 +1,105 @@
 # Changelog
 
+## Unreleased
+
+## 5.3.6 — 2026-06
+
+### Fixed
+
+- **A timeout on robots.txt / sitemap stops the sitemap phase of the host (1.6, daxx.com).** A down origin behind
+  Cloudflare does not always answer 52x: robots.txt came back after ~20 s and every sitemap request hung until the
+  client timeout, then the default candidates (`sitemap.xml`, `sitemap_index.xml`) and the driver's retries/tier
+  escalation repeated it (80–180 s per domain). Now:
+  - `SitemapParser._fetch` bounds the whole request by `SitemapParser.timeout` (covers driver retries and tier
+    escalation); a timeout is an answer "origin does not respond", not a reason to try again;
+  - the first timeout (no status + timeout error, incl. the driver's `tier timeout after …`) or CF 520–527 marks the
+    host `ORIGIN_DOWN` (`SitemapParser.mark_origin_down` / `origin_down_reason`); no further request to that host in
+    this sitemap phase — no default candidates, no nested sitemaps, no retries;
+  - `result["origin_down_reason"]` = `"timeout"` | `"cf_52x"`; `SitemapSpider.origin_down`,
+    `get_stats()["origin_down"]`, `SITEMAP_CRAWL_COMPLETED.data["origin_down"]`, and the robots.txt node gets
+    `error_message = "origin_down: <reason>"`. A spider-level `SITEMAP_PARSE_TIMEOUT` also marks the host down.
+  - Non-timeout network errors (DNS, connection refused) keep the old behaviour (default candidates are tried).
+  Live: daxx.com sitemap phase 80 s → 50 s (robots ~20 s + one 30 s timeout), healthy sites unchanged.
+- Tests: `tests/test_sitemap_origin_down_timeout.py` (8).
+
+## 5.3.5 — 2026-06
+
+### Fixed
+
+- **Dirty `Sitemap:` value in robots.txt no longer loses the sitemap (P19).** `SitemapParser._parse_robots_txt`
+  took the whole directive value verbatim (`line[8:].strip()`), so `Sitemap: sitemap URL https://site.com/sitemap_index.xml`
+  was glued into a broken address and the sitemap was lost. It now extracts the first `https?://\S+` from the value;
+  `urljoin` is applied only when the value is a relative path starting with `/`. New `SitemapParser._clean_sitemap_value`.
+- **A 404/410 is never a bot block (P19 side-finding).** `escalation._classify` returns `BotBlockStatus.NONE` for
+  status 404/410 right after the authoritative challenge headers. A dead sitemap candidate (or any not-found page) is
+  no longer classified as `captcha`/`challenge` and dragged through every tier (http→tls→browser). A real challenge
+  uses 403/503, not 404, so the authoritative header check still wins. Invariant: 404 is a definitive "not found"
+  from the server, not a page served instead of content. (After N3, verify & lock with a test — see Parser side.)
+- **A redirect inside the same ATS no longer blocks `domain_filter` (P21).** `CrawlCoordinator._handle_redirect_after_scan`
+  always allows a redirect to the **same host** (short ATS forms `apply.workable.com/j/X` → `apply.workable.com/<slug>/j/X`).
+  For a different host it still asks `domain_filter.is_allowed`, and when that refuses it consults a new host-alias
+  extension point `DomainFilter.allow_redirect(source, target)` (driven by a settable `redirect_host_alias` hook) — so
+  the Parser can pass its `DOMAIN_POLICIES` and allow `boards.greenhouse.io` → `job-boards.greenhouse.io` without
+  opening the whole internet.
+- **Sitemap under a Cloudflare 52x origin error stops without retries (1.6).** On the first `520`–`527` response to
+  robots.txt / sitemap, `SitemapParser` stops the sitemap phase and returns `result["origin_down"] = True` (logged as
+  `ORIGIN_DOWN`) instead of burning 100–180 s on retries/escalation against a down origin. New
+  `SitemapParser._is_origin_down`; `origin_down` is propagated through robots / default / nested sitemap paths.
+- **Nodes evicted by low-memory eviction are no longer lost in a browser full rescan (P23).**
+  `Graph.iter_nodes_async(include_evicted=True)` now also yields nodes that were evicted to disk (read-only, without
+  pulling them back into RAM), so a merge that iterates the source graph sees them. `Node.pinned` + `Graph.pin_node()`
+  / `Graph.pin_seed_nodes()` keep seed (`depth == 0`) vacancies in RAM — they were the oldest and got evicted first.
+  `Graph.eviction_stats` exposes the graph's own eviction counters (a backend-less graph evicts to LMDB/SQLite, which
+  `backend.get_stats()` reported as 0).
+- **A 403 storm without vendor markers is treated as a block (P24).** `EscalatingDriver` tracks a per-host window of
+  recent HTTP statuses; if a host that recently returned 2xx then returns 403 with no anti-bot markers and either ≥3
+  consecutive 403 or >30 % 403 in the window, the 403 is reclassified as `SOFT_BLOCK` (`escalation_reason=http_403_storm`)
+  and the tier escalates. A single 403 on a 2xx background is still treated as auth/method (no escalation). New
+  `_HostStatusWindow`.
+- **The HTML enrichment pool is priority-aware (2.1 / §15.4).** `run_in_html_pool(fn, *args, priority=...)` admits the
+  highest-priority waiter first when the pool is saturated (job-URLs and career subdomains enrich before catalog/blog
+  pages); with a free slot and empty queue it behaves exactly like the old FIFO semaphore. `Node` passes its URL-rule
+  priority. New `_PrioritySemaphore`.
+- **Cloudflare 520–527 is never a bot block.** `escalation._classify` returns `NONE` for 520–527 (as for 404/410):
+  the CF error page embeds `challenge-platform`, so a down origin was classified `challenge` and escalated
+  http→tls→browser (~20 s per tier) before `SitemapParser` could stop on `ORIGIN_DOWN`.
+- **robots.txt after a domain move keeps the new host's sitemap.** The same-origin gate and relative `Sitemap:`
+  paths now use the host that actually served robots.txt (`final_url`): `kilo.health` → `kilo.co/robots.txt` with
+  `Sitemap: … https://kilo.co/sitemap_index.xml` was dropped as cross-origin even after P19 cleaning.
+- **Cancelling a call waiting in the HTML-pool queue resumes the enrich budget.** With the priority semaphore the
+  acquire moved out of the `try`, so a cancellation in the queue left the node deadline suspended.
+- **`Node.pinned` does not leak into `user_data['_custom_fields']`** (`NodeMapper`, SQLite eviction storage base
+  fields) — it broke DTO/user-data invariants (`test_reserved_user_data_keys`, `test_node_mapper_content_depth`).
+- Tests: `tests/test_v535_fixes.py` (28, offline: P19, 404/52x, 1.6, P21 incl. end-to-end `async_crawl`, P23, P24, 2.1).
+
+### Added
+
+- `crawl` / `async_crawl` / `GraphCrawlerClient.crawl` / `CrawlerConfig`: `redirect_host_alias=` —
+  `Callable[[source_url, target_url], bool]` installed on the spider's `DomainFilter.redirect_host_alias`
+  (the extension point was otherwise unreachable through the public API).
+- `SitemapParser._clean_sitemap_value`, `SitemapParser._is_origin_down`; `result["origin_down"]` key.
+- `DomainFilter.allow_redirect(source_url, target_url)` and settable `DomainFilter.redirect_host_alias` hook.
+- `Node.pinned`; `Graph.pin_node`, `Graph.pin_seed_nodes`, `Graph.eviction_stats`; `iter_nodes_async(include_evicted=)`.
+- `run_in_html_pool(..., priority=)`.
+
+### Fixed (carried from Unreleased)
+
+- **Sitemap `<lastmod>` is no longer dropped.** `SitemapParser` reads `<lastmod>` of every `<url>` (with or without
+  namespace) into `result["lastmod"]` (`{url: value}`, merged across robots/default/nested sitemaps), and
+  `SitemapProcessor.create_url_nodes(lastmod=...)` stores it on the new `SitemapNode.lastmod` field (W3C datetime as
+  in the file). Consumers can now tell a 2021 post from a fresh page; `_parse_urlset` keeps its contract.
+
+- **An embedded captcha widget on a page with content is not a bot block.** `classify_block` follows the same
+  principle as the browser `CaptchaDetectorPlugin`: a block is a page served *instead of* content. After the
+  authoritative challenge headers (`cf-mitigated`, `x-amzn-waf-action`, `x-vercel-mitigated`), a non-blocking
+  response whose `<body>` has ≥1000 visible characters or ≥15 links is `NONE`, whatever widget or protection
+  script it embeds. Previously visible text was measured over the first 64 KB of the document (in WordPress that is
+  almost all `<head>`), and `cf-turnstile` / `challenge-platform` (Cloudflare loads its JSD script on regular pages)
+  were unconditional markers — so every page with reCAPTCHA/hCaptcha/Turnstile in a contact or apply form was
+  `captcha`, escalated HTTP→TLS and pinned the whole host to the TLS tier. `cf-turnstile`, `captcha-container`,
+  `challenge-platform`, `_incapsula_resource`, `awswafintegration` now count only on pages without content or with a
+  blocking status; markers and text are read from `<body>` up to 1 MB.
+
 ## 5.3.4 — 2026-10
 
 ### Fixed

{graph_crawler-5.3.4/graph_crawler.egg-info → graph_crawler-5.3.6}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: graph-crawler
-Version: 5.3.4
+Version: 5.3.6
 Summary: Web crawler that builds a graph of a site. HTTP → TLS → browser escalation with pluggable browser engines.
 Author: 0-EternalJunior-0
 Maintainer: 0-EternalJunior-0

{graph_crawler-5.3.4 → graph_crawler-5.3.6}/graph_crawler/api/_core.py

@@ -68,6 +68,7 @@ async def async_crawl_impl(
     worker_pool_size: int | None = None,  # A_liffe L3
     per_host_max_in_flight: int | None = None,  # ліміт сторінок на хост (None = без ліміту)
     node_enrich_timeout: float | None = None,  # per-node таймаут збагачення, с (None = 45 с)
+    redirect_host_alias: Any = None,  # Callable[[src, dst], bool] — дозволені крос-хост редіректи
     driver: DriverType | None = None,
     driver_config: dict[str, Any] | None = None,
     storage: StorageType | None = None,
@@ -243,6 +244,7 @@ async def async_crawl_impl(
             worker_pool_size=worker_pool_size,
             per_host_max_in_flight=per_host_max_in_flight,
             node_enrich_timeout=node_enrich_timeout,
+            redirect_host_alias=redirect_host_alias,
             # Low-memory режим
             low_memory_mode=low_memory_mode,
             evict_threshold=evict_threshold,

{graph_crawler-5.3.4 → graph_crawler-5.3.6}/graph_crawler/api/async_.py

@@ -56,6 +56,7 @@ async def async_crawl(
     worker_pool_size: int | None = None,  # A_liffe L3
     per_host_max_in_flight: int | None = None,
     node_enrich_timeout: float | None = None,
+    redirect_host_alias: Any = None,
     driver: DriverType | None = None,
     driver_config: dict[str, Any] | None = None,
     storage: StorageType | None = None,
@@ -134,6 +135,7 @@ async def async_crawl(
         worker_pool_size=worker_pool_size,
         per_host_max_in_flight=per_host_max_in_flight,
         node_enrich_timeout=node_enrich_timeout,
+        redirect_host_alias=redirect_host_alias,
         driver=driver,
         driver_config=driver_config,
         storage=storage,

{graph_crawler-5.3.4 → graph_crawler-5.3.6}/graph_crawler/api/client/client.py

@@ -130,6 +130,8 @@ class GraphCrawlerClient:
         per_host_max_in_flight: int | None = None,
         # Per-node таймаут збагачення, с (None = DEFAULT_NODE_ENRICH_TIMEOUT)
         node_enrich_timeout: float | None = None,
+        # Callable[[source_url, target_url], bool] — дозволені крос-хост редіректи
+        redirect_host_alias: Any = None,
         # Low-memory режим
         low_memory_mode: bool = False,
         evict_threshold: int = 500,
@@ -195,6 +197,7 @@ class GraphCrawlerClient:
             worker_pool_size=worker_pool_size,
             per_host_max_in_flight=per_host_max_in_flight,
             node_enrich_timeout=node_enrich_timeout,
+            redirect_host_alias=redirect_host_alias,
             # Low-memory режим
             low_memory_mode=low_memory_mode,
             evict_threshold=evict_threshold,

{graph_crawler-5.3.4 → graph_crawler-5.3.6}/graph_crawler/api/sync.py

@@ -50,6 +50,7 @@ def crawl(
     worker_pool_size: int | None = None,
     per_host_max_in_flight: int | None = None,
     node_enrich_timeout: float | None = None,
+    redirect_host_alias: Any = None,
     driver: DriverType | None = None,
     driver_config: dict[str, Any] | None = None,
     storage: StorageType | None = None,
@@ -98,6 +99,8 @@ def crawl(
         per_host_max_in_flight: ліміт одночасних запитів на хост (``None`` — без ліміту).
         node_enrich_timeout: ліміт збагачення однієї ноди, с (``None`` — 45 с); при перевищенні
             нода стає ERROR, лінки, витягнуті до таймауту, лишаються в графі.
+        redirect_host_alias: ``Callable[[source_url, target_url], bool]`` — дозволяє редірект
+            на хост поза ``allowed_domains`` (напр. між хостами однієї ATS); ``None`` — ні.
         driver: драйвер завантаження (``"http"``, ``"playwright"``, ``"escalating"`` тощо).
         driver_config: налаштування драйвера.
         storage: тип сховища (``"memory"``, ``"json"``, ``"sqlite"`` тощо).
@@ -178,6 +181,7 @@ def crawl(
         "worker_pool_size": worker_pool_size,
         "per_host_max_in_flight": per_host_max_in_flight,
         "node_enrich_timeout": node_enrich_timeout,
+        "redirect_host_alias": redirect_host_alias,
         "driver": driver,
         "driver_config": driver_config,
         "storage": storage,

{graph_crawler-5.3.4 → graph_crawler-5.3.6}/graph_crawler/application/dto/mappers/node_mapper.py

@@ -59,6 +59,7 @@ class NodeMapper:
             "processed_at",
             "should_scan",
             "can_create_edges",
+            "pinned",
             "created_at",
             "metadata",
             "user_data",

{graph_crawler-5.3.4 → graph_crawler-5.3.6}/graph_crawler/application/use_cases/crawling/crawl_coordinator.py

@@ -1003,11 +1003,30 @@ class CrawlCoordinator:
         # Захист від cross-domain редіректів повз same_domain фільтр:
         # драйвер може прозоро фоловити Location: на чужий домен, тож
         # питаємо domain_filter перед тим як зафіксувати редірект у графі.
+        #
+        # P21: редірект на ТОЙ САМИЙ хост дозволяємо завжди (коротка форма ATS
+        # `apply.workable.com/j/X` → `apply.workable.com/<slug>/j/X`). Для різних
+        # хостів — domain_filter; якщо він відмовив, пробуємо точку розширення
+        # «аліаси хостів» (`allow_redirect`), якою Parser передає свої
+        # DOMAIN_POLICIES (`boards.greenhouse.io` → `job-boards.greenhouse.io`).
         try:
+            from urllib.parse import urlsplit
+
+            src_host = urlsplit(original_node.url).hostname or ""
+            tgt_host = urlsplit(final_url).hostname or ""
+            same_host = bool(src_host) and src_host == tgt_host
+
             domain_filter = getattr(self.processor, "domain_filter", None)
-            if domain_filter is not None and not domain_filter.is_allowed(
-                final_url, source_url=original_node.url
-            ):
+            blocked = False
+            if not same_host and domain_filter is not None:
+                if not domain_filter.is_allowed(final_url, source_url=original_node.url):
+                    alias_hook = getattr(domain_filter, "allow_redirect", None)
+                    allowed_by_alias = bool(
+                        alias_hook and alias_hook(original_node.url, final_url)
+                    )
+                    blocked = not allowed_by_alias
+
+            if blocked:
                 logger.info(
                     "Redirect blocked by domain_filter: %s -> %s (cross-domain or disallowed)",
                     original_node.url,

{graph_crawler-5.3.4 → graph_crawler-5.3.6}/graph_crawler/application/use_cases/crawling/filters/domain_filter.py

@@ -47,6 +47,13 @@ class DomainFilter(BaseURLFilter):
         super().__init__(config.model_dump() if hasattr(config, "model_dump") else dict(config))
         self.event_bus = event_bus
 
+        # P21: точка розширення «аліаси хостів». Callable[[source_url, target_url], bool].
+        # Коли редірект веде на хост поза allowed_domains, координатор питає цей
+        # хук — так Parser дозволяє редірект між хостами однієї ATS-політики
+        # (boards.greenhouse.io → job-boards.greenhouse.io), не відкриваючи весь
+        # інтернет. None → редірект за межі allowed_domains блокується, як раніше.
+        self.redirect_host_alias = None
+
         # Парсимо спеціальні патерни
         self._parse_special_patterns()
 
@@ -92,6 +99,30 @@ class DomainFilter(BaseURLFilter):
     def name(self) -> str:
         return "domain"
 
+    def allow_redirect(self, source_url: str, target_url: str) -> bool:
+        """Точка розширення P21: чи дозволити редірект на хост поза allowed_domains.
+
+        Координатор завжди дозволяє редірект на той самий хост сам; сюди
+        потрапляють лише крос-хостові редіректи, які не пройшли ``is_allowed``.
+        За замовчуванням ``False``; якщо встановлено ``redirect_host_alias``
+        (Parser передає свої DOMAIN_POLICIES) — делегуємо рішення йому.
+
+        Args:
+            source_url: URL, що редіректив.
+            target_url: Фінальний URL редіректу.
+
+        Returns:
+            True, якщо редірект на цей чужий хост дозволено як аліас.
+        """
+        hook = self.redirect_host_alias
+        if hook is None:
+            return False
+        try:
+            return bool(hook(source_url, target_url))
+        except Exception as exc:  # noqa: BLE001 — хук не повинен валити обхід
+            logger.warning("redirect_host_alias hook failed for %s -> %s: %s", source_url, target_url, exc)
+            return False
+
     def _is_subdomain_of(self, domain: str, base_domain: str) -> bool:
         """
         Перевіряє чи domain є субдоменом base_domain або самим base_domain.

{graph_crawler-5.3.4 → graph_crawler-5.3.6}/graph_crawler/application/use_cases/crawling/sitemap_parser.py

@@ -15,6 +15,7 @@ import asyncio
 import gzip
 import logging
 import os
+import re
 from concurrent.futures import ThreadPoolExecutor
 from dataclasses import dataclass, field
 from typing import TYPE_CHECKING
@@ -63,6 +64,12 @@ _MSG_UNKNOWN_SITEMAP_TYPE = "Unknown sitemap type: %s"
 # тому такі файли читаються напряму через aiohttp навіть за наявності драйвера.
 _BINARY_CONTENT_TYPES = ("gzip", "octet-stream")
 
+# Причини ORIGIN_DOWN: origin не відповідає — sitemap-фаза хоста зупиняється без ретраїв.
+ORIGIN_DOWN_CF_5XX = "cf_52x"
+ORIGIN_DOWN_TIMEOUT = "timeout"
+# Як різні транспорти формулюють таймаут у ``FetchResponse.error`` (aiohttp, curl_cffi, тир-ланцюг).
+_TIMEOUT_ERROR_MARKERS = ("timeout", "timed out")
+
 
 @dataclass
 class _Fetched:
@@ -153,6 +160,10 @@ class SitemapParser:
         # щоб уникнути шуму, коли той самий сайт запитується декілька разів.
         self._logged_domain_redirects: set[tuple[str, str]] = set()
 
+        # Хости з лежачим origin (CF 52x або таймаут запиту): host → причина. Перший такий
+        # сигнал зупиняє sitemap-фазу хоста — далі жодного запиту, ні ретраю, ні кандидата.
+        self._down_hosts: dict[str, str] = {}
+
         self._xml_executor = ThreadPoolExecutor(
             max_workers=4, thread_name_prefix=f"xml_parser_{id(self)}_"
         )
@@ -190,6 +201,18 @@ class SitemapParser:
         return url.lower().endswith(".gz") or any(t in content_type for t in _BINARY_CONTENT_TYPES)
 
     async def _fetch(self, url: str) -> _Fetched:
+        """robots.txt/sitemap з верхньою межею ``self.timeout`` на весь запит.
+
+        Межа покриває і ретраї, і ескалацію тирів драйвера: таймаут — це вже відповідь
+        «origin не відповідає», а не привід пробувати ще раз (див. ``ORIGIN_DOWN``).
+        """
+        try:
+            async with asyncio.timeout(self.timeout):
+                return await self._fetch_unbounded(url)
+        except TimeoutError:
+            return _Fetched(status=None, final_url=url, error=f"timeout after {self.timeout}s")
+
+    async def _fetch_unbounded(self, url: str) -> _Fetched:
         """robots.txt/sitemap через ``IDriver`` (тир-ланцюг) або власний aiohttp.
 
         ``FetchResponse.html`` — текст: бінарні ``.gz`` (за URL або Content-Type)
@@ -262,11 +285,16 @@ class SitemapParser:
                 - ``sitemap_indexes``: sitemap-index URL (якщо є).
         """
         result = {"sitemap_urls": [], "urls": [], "sitemap_indexes": []}
+        robots_url = urljoin(base_url, "/robots.txt")
+        if self._mark_if_origin_down_cached(robots_url, result):
+            return result
 
         try:
-            robots_url = urljoin(base_url, "/robots.txt")
             fetched = await self._fetch(robots_url)
 
+            if self._check_origin(robots_url, fetched):
+                return self._with_origin_down(robots_url, result)
+
             if fetched.status != 200 or fetched.body is None:
                 logger.warning("robots.txt not found at %s", robots_url)
                 return await self._try_default_sitemaps_async(base_url, result)
@@ -275,7 +303,10 @@ class SitemapParser:
             body = fetched.body
             robots_content = body if isinstance(body, str) else body.decode("utf-8", "replace")
 
-            sitemap_urls = self._parse_robots_txt(robots_content, base_url)
+            # Origin і база відносних шляхів — хост, що реально віддав robots.txt:
+            # після переїзду домену (kilo.health → kilo.co) його Sitemap: вказує на новий хост.
+            robots_base = fetched.final_url or base_url
+            sitemap_urls = self._parse_robots_txt(robots_content, robots_base)
 
             if sitemap_urls:
                 result["sitemap_urls"] = sitemap_urls
@@ -290,15 +321,20 @@ class SitemapParser:
                         logger.warning("Sitemap parse error: %s", sitemap_data)
                         continue
                     if isinstance(sitemap_data, dict):
+                        if sitemap_data.get("origin_down"):
+                            result["origin_down"] = True
+                            result["origin_down_reason"] = sitemap_data.get("origin_down_reason")
                         result["urls"].extend(sitemap_data.get("urls", []))
                         result["sitemap_indexes"].extend(sitemap_data.get("sitemap_indexes", []))
+                        result.setdefault("lastmod", {}).update(sitemap_data.get("lastmod", {}))
             else:
                 logger.warning(_MSG_SITEMAP_NOT_FOUND_IN_ROBOTS, robots_url)
                 result = await self._try_default_sitemaps_async(base_url, result)
 
         except Exception as e:
             logger.error(_MSG_ROBOTS_READ_ERROR, base_url, e)
-            result = await self._try_default_sitemaps_async(base_url, result)
+            if not self._mark_if_origin_down_cached(robots_url, result):
+                result = await self._try_default_sitemaps_async(base_url, result)
 
         result["urls"] = list(set(result["urls"]))
         result["sitemap_indexes"] = list(set(result["sitemap_indexes"]))
@@ -316,10 +352,15 @@ class SitemapParser:
         for sitemap_url in default_sitemaps:
             try:
                 sitemap_data = await self.parse_sitemap_async(sitemap_url)
+                if sitemap_data.get("origin_down"):
+                    result["origin_down"] = True
+                    result["origin_down_reason"] = sitemap_data.get("origin_down_reason")
+                    break
                 if sitemap_data["urls"] or sitemap_data["sitemap_indexes"]:
                     result["sitemap_urls"].append(sitemap_url)
                     result["urls"].extend(sitemap_data["urls"])
                     result["sitemap_indexes"].extend(sitemap_data["sitemap_indexes"])
+                    result.setdefault("lastmod", {}).update(sitemap_data.get("lastmod", {}))
                     logger.info(_MSG_SITEMAP_FOUND_AT, sitemap_url)
                     break
             except Exception as e:
@@ -346,10 +387,15 @@ class SitemapParser:
         if cached is not None:
             logger.debug("Sitemap cache hit: %s", sitemap_url)
             return cached
+        if self._mark_if_origin_down_cached(sitemap_url, result):
+            return result
 
         try:
             fetched = await self._fetch(sitemap_url)
 
+            if self._check_origin(sitemap_url, fetched):
+                return self._with_origin_down(sitemap_url, result)
+
             if fetched.status not in (200, 301, 302) or fetched.body is None:
                 logger.warning(
                     _MSG_SITEMAP_NOT_FOUND_STATUS,
@@ -439,8 +485,11 @@ class SitemapParser:
                 )
 
             elif root.tag.endswith("urlset"):
-                raw_urls = self._parse_urlset(root)
-                result["urls"] = [self._normalize_url(u, base_url) for u in raw_urls]
+                entries = self._parse_urlset_entries(root)
+                result["urls"] = [self._normalize_url(u, base_url) for u, _ in entries]
+                result["lastmod"] = {
+                    self._normalize_url(u, base_url): lastmod for u, lastmod in entries if lastmod
+                }
                 logger.info(_MSG_URLS_FOUND_IN_SITEMAP, len(result["urls"]), sitemap_url)
 
             else:
@@ -475,7 +524,7 @@ class SitemapParser:
         for line in content.splitlines():
             line = line.strip()
             if line.lower().startswith("sitemap:"):
-                url = line[8:].strip()
+                url = self._clean_sitemap_value(line[8:])
                 if not url:
                     continue
                 normalized = self._normalize_url(url, base_url)
@@ -508,6 +557,86 @@ class SitemapParser:
             return True
         return False
 
+    @staticmethod
+    def _clean_sitemap_value(raw: str) -> str | None:
+        """Витягує sitemap URL з (можливо брудного) значення директиви ``Sitemap:``.
+
+        Деякі robots.txt містять сміття перед адресою, напр.
+        ``Sitemap: sitemap URL https://site.com/sitemap_index.xml``. Беремо
+        перший ``https?://\\S+`` із значення; якщо абсолютного URL немає, але
+        значення — відносний шлях (``/sitemap.xml``), повертаємо його для
+        подальшого ``urljoin``. Інакше — ``None`` (значення непридатне).
+        """
+        raw = raw.strip()
+        if not raw:
+            return None
+        match = re.search(r"https?://\S+", raw)
+        if match:
+            return match.group(0)
+        if raw.startswith("/"):
+            return raw.split()[0]
+        return None
+
+    def origin_down_reason(self, url: str) -> str | None:
+        """Причина ORIGIN_DOWN для хоста ``url`` (``cf_52x``/``timeout``) або ``None``."""
+        return self._down_hosts.get(urlparse(url).hostname or "")
+
+    def mark_origin_down(self, url: str, reason: str, detail: str | int | None = None) -> None:
+        """Позначає origin хоста ``url`` лежачим: решта sitemap-фази хоста — без запитів."""
+        host = urlparse(url).hostname or ""
+        if not host or host in self._down_hosts:
+            return
+        self._down_hosts[host] = reason
+        logger.warning(
+            "ORIGIN_DOWN: %s (%s, %s on %s) — stopping sitemap phase for host without retries",
+            host,
+            reason,
+            detail,
+            url,
+        )
+
+    def _check_origin(self, url: str, fetched: _Fetched) -> bool:
+        """``True`` і позначка хоста, якщо відповідь каже «origin лежить» (CF 52x або таймаут)."""
+        if self._is_origin_down(fetched.status):
+            self.mark_origin_down(url, ORIGIN_DOWN_CF_5XX, fetched.status)
+            return True
+        if self._is_timeout(fetched):
+            self.mark_origin_down(url, ORIGIN_DOWN_TIMEOUT, fetched.error)
+            return True
+        return False
+
+    def _with_origin_down(self, url: str, result: dict) -> dict:
+        result["origin_down"] = True
+        result["origin_down_reason"] = self.origin_down_reason(url)
+        return result
+
+    def _mark_if_origin_down_cached(self, url: str, result: dict) -> bool:
+        """Хост уже позначено лежачим — віддаємо ``origin_down`` без запиту."""
+        if self.origin_down_reason(url) is None:
+            return False
+        logger.debug("ORIGIN_DOWN: skipping %s (host already down)", url)
+        self._with_origin_down(url, result)
+        return True
+
+    @staticmethod
+    def _is_timeout(fetched: _Fetched) -> bool:
+        """Таймаут без відповіді сервера: статусу немає, у помилці — ознака таймауту."""
+        if fetched.status is not None or not fetched.error:
+            return False
+        error = fetched.error.lower()
+        return any(marker in error for marker in _TIMEOUT_ERROR_MARKERS)
+
+    @staticmethod
+    def _is_origin_down(status: int | None) -> bool:
+        """``True`` для Cloudflare-origin помилок 520–527 (origin недоступний).
+
+        520 Web Server Returned an Unknown Error … 527 Railgun Error — проксі
+        Cloudflare не зміг дістатися origin. Ретраї/ескалація на robots/sitemap
+        безнадійні: origin лежить, а не блокує. Зупиняємо sitemap-фазу й віддаємо
+        сигнал ``ORIGIN_DOWN`` замість марних 100–180 с ретраїв.
+        """
+        return status is not None and 520 <= status <= 527
+
     def parse_from_robots(self, base_url: str) -> dict[str, list[str]]:
         """Sync-обгортка над ``parse_from_robots_async`` (``asyncio.run``).
 
@@ -555,18 +684,25 @@ class SitemapParser:
 
     def _parse_urlset(self, root: Element) -> list[str]:
         """Повертає URL з urlset-документа (з namespace або без)."""
-        urls = []
+        return [url for url, _ in self._parse_urlset_entries(root)]
 
-        url_elements = root.findall(".//sm:url/sm:loc", self.SITEMAP_NS)
+    def _parse_urlset_entries(self, root: Element) -> list[tuple[str, str | None]]:
+        """``(loc, lastmod)`` кожного ``<url>`` urlset-документа (з namespace або без)."""
+        ns = self.SITEMAP_NS
+        url_elements = root.findall(".//sm:url", ns)
+        prefix = "sm:" if url_elements else ""
         if not url_elements:
-            url_elements = root.findall(".//url/loc")
+            url_elements = root.findall(".//url")
 
+        entries = []
         for url_elem in url_elements:
-            url = url_elem.text
-            if url:
-                urls.append(url.strip())
-
-        return urls
+            loc = url_elem.find(f"{prefix}loc", ns)
+            if loc is None or not loc.text or not loc.text.strip():
+                continue
+            lastmod = url_elem.find(f"{prefix}lastmod", ns)
+            value = lastmod.text.strip() if lastmod is not None and lastmod.text else None
+            entries.append((loc.text.strip(), value or None))
+        return entries
 
     def parse_large_sitemap_streaming(self, filepath_or_content: str | bytes) -> list[str]:
         """Потоковий парсинг великих sitemap (пам'ять не залежить від розміру файлу).

{graph_crawler-5.3.4 → graph_crawler-5.3.6}/graph_crawler/application/use_cases/crawling/sitemap_processor.py

@@ -209,6 +209,7 @@ class SitemapProcessor:
         parent_sitemap_url: str,
         depth: int = 2,
         max_urls: int | None = None,
+        lastmod: dict[str, str] | None = None,
     ) -> list[SitemapNode]:
         """
         Створює Node для кожного URL зі sitemap.
@@ -218,6 +219,7 @@ class SitemapProcessor:
             parent_sitemap_url: URL батьківського sitemap
             depth: Глибина у графі
             max_urls: Максимальна кількість URL для обробки (None = всі)
+            lastmod: ``{url: <lastmod>}`` з того ж sitemap → ``SitemapNode.lastmod``
 
         Returns:
             Список створених SitemapNode
@@ -261,6 +263,7 @@ class SitemapProcessor:
                 url=url,
                 node_type="url",
                 parent_sitemap=parent_sitemap_url,
+                lastmod=(lastmod or {}).get(url),
                 depth=depth,
                 should_scan=False,  # URL вузли не сканують (це кінцеві точки)
                 can_create_edges=False,