graph-crawler 5.3.4__tar.gz → 5.3.5__tar.gz

{graph_crawler-5.3.4 → graph_crawler-5.3.5}/CHANGELOG.md

@@ -1,5 +1,85 @@
 # Changelog
 
+## Unreleased
+
+## 5.3.5 — 2026-06
+
+### Fixed
+
+- **Dirty `Sitemap:` value in robots.txt no longer loses the sitemap (P19).** `SitemapParser._parse_robots_txt`
+  took the whole directive value verbatim (`line[8:].strip()`), so `Sitemap: sitemap URL https://site.com/sitemap_index.xml`
+  was glued into a broken address and the sitemap was lost. It now extracts the first `https?://\S+` from the value;
+  `urljoin` is applied only when the value is a relative path starting with `/`. New `SitemapParser._clean_sitemap_value`.
+- **A 404/410 is never a bot block (P19 side-finding).** `escalation._classify` returns `BotBlockStatus.NONE` for
+  status 404/410 right after the authoritative challenge headers. A dead sitemap candidate (or any not-found page) is
+  no longer classified as `captcha`/`challenge` and dragged through every tier (http→tls→browser). A real challenge
+  uses 403/503, not 404, so the authoritative header check still wins. Invariant: 404 is a definitive "not found"
+  from the server, not a page served instead of content. (After N3, verify & lock with a test — see Parser side.)
+- **A redirect inside the same ATS no longer blocks `domain_filter` (P21).** `CrawlCoordinator._handle_redirect_after_scan`
+  always allows a redirect to the **same host** (short ATS forms `apply.workable.com/j/X` → `apply.workable.com/<slug>/j/X`).
+  For a different host it still asks `domain_filter.is_allowed`, and when that refuses it consults a new host-alias
+  extension point `DomainFilter.allow_redirect(source, target)` (driven by a settable `redirect_host_alias` hook) — so
+  the Parser can pass its `DOMAIN_POLICIES` and allow `boards.greenhouse.io` → `job-boards.greenhouse.io` without
+  opening the whole internet.
+- **Sitemap under a Cloudflare 52x origin error stops without retries (1.6).** On the first `520`–`527` response to
+  robots.txt / sitemap, `SitemapParser` stops the sitemap phase and returns `result["origin_down"] = True` (logged as
+  `ORIGIN_DOWN`) instead of burning 100–180 s on retries/escalation against a down origin. New
+  `SitemapParser._is_origin_down`; `origin_down` is propagated through robots / default / nested sitemap paths.
+- **Nodes evicted by low-memory eviction are no longer lost in a browser full rescan (P23).**
+  `Graph.iter_nodes_async(include_evicted=True)` now also yields nodes that were evicted to disk (read-only, without
+  pulling them back into RAM), so a merge that iterates the source graph sees them. `Node.pinned` + `Graph.pin_node()`
+  / `Graph.pin_seed_nodes()` keep seed (`depth == 0`) vacancies in RAM — they were the oldest and got evicted first.
+  `Graph.eviction_stats` exposes the graph's own eviction counters (a backend-less graph evicts to LMDB/SQLite, which
+  `backend.get_stats()` reported as 0).
+- **A 403 storm without vendor markers is treated as a block (P24).** `EscalatingDriver` tracks a per-host window of
+  recent HTTP statuses; if a host that recently returned 2xx then returns 403 with no anti-bot markers and either ≥3
+  consecutive 403 or >30 % 403 in the window, the 403 is reclassified as `SOFT_BLOCK` (`escalation_reason=http_403_storm`)
+  and the tier escalates. A single 403 on a 2xx background is still treated as auth/method (no escalation). New
+  `_HostStatusWindow`.
+- **The HTML enrichment pool is priority-aware (2.1 / §15.4).** `run_in_html_pool(fn, *args, priority=...)` admits the
+  highest-priority waiter first when the pool is saturated (job-URLs and career subdomains enrich before catalog/blog
+  pages); with a free slot and empty queue it behaves exactly like the old FIFO semaphore. `Node` passes its URL-rule
+  priority. New `_PrioritySemaphore`.
+- **Cloudflare 520–527 is never a bot block.** `escalation._classify` returns `NONE` for 520–527 (as for 404/410):
+  the CF error page embeds `challenge-platform`, so a down origin was classified `challenge` and escalated
+  http→tls→browser (~20 s per tier) before `SitemapParser` could stop on `ORIGIN_DOWN`.
+- **robots.txt after a domain move keeps the new host's sitemap.** The same-origin gate and relative `Sitemap:`
+  paths now use the host that actually served robots.txt (`final_url`): `kilo.health` → `kilo.co/robots.txt` with
+  `Sitemap: … https://kilo.co/sitemap_index.xml` was dropped as cross-origin even after P19 cleaning.
+- **Cancelling a call waiting in the HTML-pool queue resumes the enrich budget.** With the priority semaphore the
+  acquire moved out of the `try`, so a cancellation in the queue left the node deadline suspended.
+- **`Node.pinned` does not leak into `user_data['_custom_fields']`** (`NodeMapper`, SQLite eviction storage base
+  fields) — it broke DTO/user-data invariants (`test_reserved_user_data_keys`, `test_node_mapper_content_depth`).
+- Tests: `tests/test_v535_fixes.py` (28, offline: P19, 404/52x, 1.6, P21 incl. end-to-end `async_crawl`, P23, P24, 2.1).
+
+### Added
+
+- `crawl` / `async_crawl` / `GraphCrawlerClient.crawl` / `CrawlerConfig`: `redirect_host_alias=` —
+  `Callable[[source_url, target_url], bool]` installed on the spider's `DomainFilter.redirect_host_alias`
+  (the extension point was otherwise unreachable through the public API).
+- `SitemapParser._clean_sitemap_value`, `SitemapParser._is_origin_down`; `result["origin_down"]` key.
+- `DomainFilter.allow_redirect(source_url, target_url)` and settable `DomainFilter.redirect_host_alias` hook.
+- `Node.pinned`; `Graph.pin_node`, `Graph.pin_seed_nodes`, `Graph.eviction_stats`; `iter_nodes_async(include_evicted=)`.
+- `run_in_html_pool(..., priority=)`.
+
+### Fixed (carried from Unreleased)
+
+- **Sitemap `<lastmod>` is no longer dropped.** `SitemapParser` reads `<lastmod>` of every `<url>` (with or without
+  namespace) into `result["lastmod"]` (`{url: value}`, merged across robots/default/nested sitemaps), and
+  `SitemapProcessor.create_url_nodes(lastmod=...)` stores it on the new `SitemapNode.lastmod` field (W3C datetime as
+  in the file). Consumers can now tell a 2021 post from a fresh page; `_parse_urlset` keeps its contract.
+
+- **An embedded captcha widget on a page with content is not a bot block.** `classify_block` follows the same
+  principle as the browser `CaptchaDetectorPlugin`: a block is a page served *instead of* content. After the
+  authoritative challenge headers (`cf-mitigated`, `x-amzn-waf-action`, `x-vercel-mitigated`), a non-blocking
+  response whose `<body>` has ≥1000 visible characters or ≥15 links is `NONE`, whatever widget or protection
+  script it embeds. Previously visible text was measured over the first 64 KB of the document (in WordPress that is
+  almost all `<head>`), and `cf-turnstile` / `challenge-platform` (Cloudflare loads its JSD script on regular pages)
+  were unconditional markers — so every page with reCAPTCHA/hCaptcha/Turnstile in a contact or apply form was
+  `captcha`, escalated HTTP→TLS and pinned the whole host to the TLS tier. `cf-turnstile`, `captcha-container`,
+  `challenge-platform`, `_incapsula_resource`, `awswafintegration` now count only on pages without content or with a
+  blocking status; markers and text are read from `<body>` up to 1 MB.
+
 ## 5.3.4 — 2026-10
 
 ### Fixed

{graph_crawler-5.3.4/graph_crawler.egg-info → graph_crawler-5.3.5}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: graph-crawler
-Version: 5.3.4
+Version: 5.3.5
 Summary: Web crawler that builds a graph of a site. HTTP → TLS → browser escalation with pluggable browser engines.
 Author: 0-EternalJunior-0
 Maintainer: 0-EternalJunior-0

{graph_crawler-5.3.4 → graph_crawler-5.3.5}/graph_crawler/api/_core.py

@@ -68,6 +68,7 @@ async def async_crawl_impl(
     worker_pool_size: int | None = None,  # A_liffe L3
     per_host_max_in_flight: int | None = None,  # ліміт сторінок на хост (None = без ліміту)
     node_enrich_timeout: float | None = None,  # per-node таймаут збагачення, с (None = 45 с)
+    redirect_host_alias: Any = None,  # Callable[[src, dst], bool] — дозволені крос-хост редіректи
     driver: DriverType | None = None,
     driver_config: dict[str, Any] | None = None,
     storage: StorageType | None = None,
@@ -243,6 +244,7 @@ async def async_crawl_impl(
             worker_pool_size=worker_pool_size,
             per_host_max_in_flight=per_host_max_in_flight,
             node_enrich_timeout=node_enrich_timeout,
+            redirect_host_alias=redirect_host_alias,
             # Low-memory режим
             low_memory_mode=low_memory_mode,
             evict_threshold=evict_threshold,

{graph_crawler-5.3.4 → graph_crawler-5.3.5}/graph_crawler/api/async_.py

@@ -56,6 +56,7 @@ async def async_crawl(
     worker_pool_size: int | None = None,  # A_liffe L3
     per_host_max_in_flight: int | None = None,
     node_enrich_timeout: float | None = None,
+    redirect_host_alias: Any = None,
     driver: DriverType | None = None,
     driver_config: dict[str, Any] | None = None,
     storage: StorageType | None = None,
@@ -134,6 +135,7 @@ async def async_crawl(
         worker_pool_size=worker_pool_size,
         per_host_max_in_flight=per_host_max_in_flight,
         node_enrich_timeout=node_enrich_timeout,
+        redirect_host_alias=redirect_host_alias,
         driver=driver,
         driver_config=driver_config,
         storage=storage,

{graph_crawler-5.3.4 → graph_crawler-5.3.5}/graph_crawler/api/client/client.py

@@ -130,6 +130,8 @@ class GraphCrawlerClient:
         per_host_max_in_flight: int | None = None,
         # Per-node таймаут збагачення, с (None = DEFAULT_NODE_ENRICH_TIMEOUT)
         node_enrich_timeout: float | None = None,
+        # Callable[[source_url, target_url], bool] — дозволені крос-хост редіректи
+        redirect_host_alias: Any = None,
         # Low-memory режим
         low_memory_mode: bool = False,
         evict_threshold: int = 500,
@@ -195,6 +197,7 @@ class GraphCrawlerClient:
             worker_pool_size=worker_pool_size,
             per_host_max_in_flight=per_host_max_in_flight,
             node_enrich_timeout=node_enrich_timeout,
+            redirect_host_alias=redirect_host_alias,
             # Low-memory режим
             low_memory_mode=low_memory_mode,
             evict_threshold=evict_threshold,

{graph_crawler-5.3.4 → graph_crawler-5.3.5}/graph_crawler/api/sync.py

@@ -50,6 +50,7 @@ def crawl(
     worker_pool_size: int | None = None,
     per_host_max_in_flight: int | None = None,
     node_enrich_timeout: float | None = None,
+    redirect_host_alias: Any = None,
     driver: DriverType | None = None,
     driver_config: dict[str, Any] | None = None,
     storage: StorageType | None = None,
@@ -98,6 +99,8 @@ def crawl(
         per_host_max_in_flight: ліміт одночасних запитів на хост (``None`` — без ліміту).
         node_enrich_timeout: ліміт збагачення однієї ноди, с (``None`` — 45 с); при перевищенні
             нода стає ERROR, лінки, витягнуті до таймауту, лишаються в графі.
+        redirect_host_alias: ``Callable[[source_url, target_url], bool]`` — дозволяє редірект
+            на хост поза ``allowed_domains`` (напр. між хостами однієї ATS); ``None`` — ні.
         driver: драйвер завантаження (``"http"``, ``"playwright"``, ``"escalating"`` тощо).
         driver_config: налаштування драйвера.
         storage: тип сховища (``"memory"``, ``"json"``, ``"sqlite"`` тощо).
@@ -178,6 +181,7 @@ def crawl(
         "worker_pool_size": worker_pool_size,
         "per_host_max_in_flight": per_host_max_in_flight,
         "node_enrich_timeout": node_enrich_timeout,
+        "redirect_host_alias": redirect_host_alias,
         "driver": driver,
         "driver_config": driver_config,
         "storage": storage,

{graph_crawler-5.3.4 → graph_crawler-5.3.5}/graph_crawler/application/dto/mappers/node_mapper.py

@@ -59,6 +59,7 @@ class NodeMapper:
             "processed_at",
             "should_scan",
             "can_create_edges",
+            "pinned",
             "created_at",
             "metadata",
             "user_data",

{graph_crawler-5.3.4 → graph_crawler-5.3.5}/graph_crawler/application/use_cases/crawling/crawl_coordinator.py

@@ -1003,11 +1003,30 @@ class CrawlCoordinator:
         # Захист від cross-domain редіректів повз same_domain фільтр:
         # драйвер може прозоро фоловити Location: на чужий домен, тож
         # питаємо domain_filter перед тим як зафіксувати редірект у графі.
+        #
+        # P21: редірект на ТОЙ САМИЙ хост дозволяємо завжди (коротка форма ATS
+        # `apply.workable.com/j/X` → `apply.workable.com/<slug>/j/X`). Для різних
+        # хостів — domain_filter; якщо він відмовив, пробуємо точку розширення
+        # «аліаси хостів» (`allow_redirect`), якою Parser передає свої
+        # DOMAIN_POLICIES (`boards.greenhouse.io` → `job-boards.greenhouse.io`).
         try:
+            from urllib.parse import urlsplit
+
+            src_host = urlsplit(original_node.url).hostname or ""
+            tgt_host = urlsplit(final_url).hostname or ""
+            same_host = bool(src_host) and src_host == tgt_host
+
             domain_filter = getattr(self.processor, "domain_filter", None)
-            if domain_filter is not None and not domain_filter.is_allowed(
-                final_url, source_url=original_node.url
-            ):
+            blocked = False
+            if not same_host and domain_filter is not None:
+                if not domain_filter.is_allowed(final_url, source_url=original_node.url):
+                    alias_hook = getattr(domain_filter, "allow_redirect", None)
+                    allowed_by_alias = bool(
+                        alias_hook and alias_hook(original_node.url, final_url)
+                    )
+                    blocked = not allowed_by_alias
+
+            if blocked:
                 logger.info(
                     "Redirect blocked by domain_filter: %s -> %s (cross-domain or disallowed)",
                     original_node.url,

{graph_crawler-5.3.4 → graph_crawler-5.3.5}/graph_crawler/application/use_cases/crawling/filters/domain_filter.py

@@ -47,6 +47,13 @@ class DomainFilter(BaseURLFilter):
         super().__init__(config.model_dump() if hasattr(config, "model_dump") else dict(config))
         self.event_bus = event_bus
 
+        # P21: точка розширення «аліаси хостів». Callable[[source_url, target_url], bool].
+        # Коли редірект веде на хост поза allowed_domains, координатор питає цей
+        # хук — так Parser дозволяє редірект між хостами однієї ATS-політики
+        # (boards.greenhouse.io → job-boards.greenhouse.io), не відкриваючи весь
+        # інтернет. None → редірект за межі allowed_domains блокується, як раніше.
+        self.redirect_host_alias = None
+
         # Парсимо спеціальні патерни
         self._parse_special_patterns()
 
@@ -92,6 +99,30 @@ class DomainFilter(BaseURLFilter):
     def name(self) -> str:
         return "domain"
 
+    def allow_redirect(self, source_url: str, target_url: str) -> bool:
+        """Точка розширення P21: чи дозволити редірект на хост поза allowed_domains.
+
+        Координатор завжди дозволяє редірект на той самий хост сам; сюди
+        потрапляють лише крос-хостові редіректи, які не пройшли ``is_allowed``.
+        За замовчуванням ``False``; якщо встановлено ``redirect_host_alias``
+        (Parser передає свої DOMAIN_POLICIES) — делегуємо рішення йому.
+
+        Args:
+            source_url: URL, що редіректив.
+            target_url: Фінальний URL редіректу.
+
+        Returns:
+            True, якщо редірект на цей чужий хост дозволено як аліас.
+        """
+        hook = self.redirect_host_alias
+        if hook is None:
+            return False
+        try:
+            return bool(hook(source_url, target_url))
+        except Exception as exc:  # noqa: BLE001 — хук не повинен валити обхід
+            logger.warning("redirect_host_alias hook failed for %s -> %s: %s", source_url, target_url, exc)
+            return False
+
     def _is_subdomain_of(self, domain: str, base_domain: str) -> bool:
         """
         Перевіряє чи domain є субдоменом base_domain або самим base_domain.

{graph_crawler-5.3.4 → graph_crawler-5.3.5}/graph_crawler/application/use_cases/crawling/sitemap_parser.py

@@ -15,6 +15,7 @@ import asyncio
 import gzip
 import logging
 import os
+import re
 from concurrent.futures import ThreadPoolExecutor
 from dataclasses import dataclass, field
 from typing import TYPE_CHECKING
@@ -267,6 +268,16 @@ class SitemapParser:
             robots_url = urljoin(base_url, "/robots.txt")
             fetched = await self._fetch(robots_url)
 
+            if self._is_origin_down(fetched.status):
+                logger.warning(
+                    "ORIGIN_DOWN: %s returned %s (Cloudflare origin error) — "
+                    "stopping sitemap phase without retries",
+                    robots_url,
+                    fetched.status,
+                )
+                result["origin_down"] = True
+                return result
+
             if fetched.status != 200 or fetched.body is None:
                 logger.warning("robots.txt not found at %s", robots_url)
                 return await self._try_default_sitemaps_async(base_url, result)
@@ -275,7 +286,10 @@ class SitemapParser:
             body = fetched.body
             robots_content = body if isinstance(body, str) else body.decode("utf-8", "replace")
 
-            sitemap_urls = self._parse_robots_txt(robots_content, base_url)
+            # Origin і база відносних шляхів — хост, що реально віддав robots.txt:
+            # після переїзду домену (kilo.health → kilo.co) його Sitemap: вказує на новий хост.
+            robots_base = fetched.final_url or base_url
+            sitemap_urls = self._parse_robots_txt(robots_content, robots_base)
 
             if sitemap_urls:
                 result["sitemap_urls"] = sitemap_urls
@@ -290,8 +304,11 @@ class SitemapParser:
                         logger.warning("Sitemap parse error: %s", sitemap_data)
                         continue
                     if isinstance(sitemap_data, dict):
+                        if sitemap_data.get("origin_down"):
+                            result["origin_down"] = True
                         result["urls"].extend(sitemap_data.get("urls", []))
                         result["sitemap_indexes"].extend(sitemap_data.get("sitemap_indexes", []))
+                        result.setdefault("lastmod", {}).update(sitemap_data.get("lastmod", {}))
             else:
                 logger.warning(_MSG_SITEMAP_NOT_FOUND_IN_ROBOTS, robots_url)
                 result = await self._try_default_sitemaps_async(base_url, result)
@@ -316,10 +333,14 @@ class SitemapParser:
         for sitemap_url in default_sitemaps:
             try:
                 sitemap_data = await self.parse_sitemap_async(sitemap_url)
+                if sitemap_data.get("origin_down"):
+                    result["origin_down"] = True
+                    break
                 if sitemap_data["urls"] or sitemap_data["sitemap_indexes"]:
                     result["sitemap_urls"].append(sitemap_url)
                     result["urls"].extend(sitemap_data["urls"])
                     result["sitemap_indexes"].extend(sitemap_data["sitemap_indexes"])
+                    result.setdefault("lastmod", {}).update(sitemap_data.get("lastmod", {}))
                     logger.info(_MSG_SITEMAP_FOUND_AT, sitemap_url)
                     break
             except Exception as e:
@@ -350,6 +371,16 @@ class SitemapParser:
         try:
             fetched = await self._fetch(sitemap_url)
 
+            if self._is_origin_down(fetched.status):
+                logger.warning(
+                    "ORIGIN_DOWN: sitemap %s returned %s (Cloudflare origin error) — "
+                    "no retries",
+                    sitemap_url,
+                    fetched.status,
+                )
+                result["origin_down"] = True
+                return result
+
             if fetched.status not in (200, 301, 302) or fetched.body is None:
                 logger.warning(
                     _MSG_SITEMAP_NOT_FOUND_STATUS,
@@ -439,8 +470,11 @@ class SitemapParser:
                 )
 
             elif root.tag.endswith("urlset"):
-                raw_urls = self._parse_urlset(root)
-                result["urls"] = [self._normalize_url(u, base_url) for u in raw_urls]
+                entries = self._parse_urlset_entries(root)
+                result["urls"] = [self._normalize_url(u, base_url) for u, _ in entries]
+                result["lastmod"] = {
+                    self._normalize_url(u, base_url): lastmod for u, lastmod in entries if lastmod
+                }
                 logger.info(_MSG_URLS_FOUND_IN_SITEMAP, len(result["urls"]), sitemap_url)
 
             else:
@@ -475,7 +509,7 @@ class SitemapParser:
         for line in content.splitlines():
             line = line.strip()
             if line.lower().startswith("sitemap:"):
-                url = line[8:].strip()
+                url = self._clean_sitemap_value(line[8:])
                 if not url:
                     continue
                 normalized = self._normalize_url(url, base_url)
@@ -508,6 +542,37 @@ class SitemapParser:
             return True
         return False
 
+    @staticmethod
+    def _clean_sitemap_value(raw: str) -> str | None:
+        """Витягує sitemap URL з (можливо брудного) значення директиви ``Sitemap:``.
+
+        Деякі robots.txt містять сміття перед адресою, напр.
+        ``Sitemap: sitemap URL https://site.com/sitemap_index.xml``. Беремо
+        перший ``https?://\\S+`` із значення; якщо абсолютного URL немає, але
+        значення — відносний шлях (``/sitemap.xml``), повертаємо його для
+        подальшого ``urljoin``. Інакше — ``None`` (значення непридатне).
+        """
+        raw = raw.strip()
+        if not raw:
+            return None
+        match = re.search(r"https?://\S+", raw)
+        if match:
+            return match.group(0)
+        if raw.startswith("/"):
+            return raw.split()[0]
+        return None
+
+    @staticmethod
+    def _is_origin_down(status: int | None) -> bool:
+        """``True`` для Cloudflare-origin помилок 520–527 (origin недоступний).
+
+        520 Web Server Returned an Unknown Error … 527 Railgun Error — проксі
+        Cloudflare не зміг дістатися origin. Ретраї/ескалація на robots/sitemap
+        безнадійні: origin лежить, а не блокує. Зупиняємо sitemap-фазу й віддаємо
+        сигнал ``ORIGIN_DOWN`` замість марних 100–180 с ретраїв.
+        """
+        return status is not None and 520 <= status <= 527
+
     def parse_from_robots(self, base_url: str) -> dict[str, list[str]]:
         """Sync-обгортка над ``parse_from_robots_async`` (``asyncio.run``).
 
@@ -555,18 +620,25 @@ class SitemapParser:
 
     def _parse_urlset(self, root: Element) -> list[str]:
         """Повертає URL з urlset-документа (з namespace або без)."""
-        urls = []
+        return [url for url, _ in self._parse_urlset_entries(root)]
 
-        url_elements = root.findall(".//sm:url/sm:loc", self.SITEMAP_NS)
+    def _parse_urlset_entries(self, root: Element) -> list[tuple[str, str | None]]:
+        """``(loc, lastmod)`` кожного ``<url>`` urlset-документа (з namespace або без)."""
+        ns = self.SITEMAP_NS
+        url_elements = root.findall(".//sm:url", ns)
+        prefix = "sm:" if url_elements else ""
         if not url_elements:
-            url_elements = root.findall(".//url/loc")
+            url_elements = root.findall(".//url")
 
+        entries = []
         for url_elem in url_elements:
-            url = url_elem.text
-            if url:
-                urls.append(url.strip())
-
-        return urls
+            loc = url_elem.find(f"{prefix}loc", ns)
+            if loc is None or not loc.text or not loc.text.strip():
+                continue
+            lastmod = url_elem.find(f"{prefix}lastmod", ns)
+            value = lastmod.text.strip() if lastmod is not None and lastmod.text else None
+            entries.append((loc.text.strip(), value or None))
+        return entries
 
     def parse_large_sitemap_streaming(self, filepath_or_content: str | bytes) -> list[str]:
         """Потоковий парсинг великих sitemap (пам'ять не залежить від розміру файлу).

{graph_crawler-5.3.4 → graph_crawler-5.3.5}/graph_crawler/application/use_cases/crawling/sitemap_processor.py

@@ -209,6 +209,7 @@ class SitemapProcessor:
         parent_sitemap_url: str,
         depth: int = 2,
         max_urls: int | None = None,
+        lastmod: dict[str, str] | None = None,
     ) -> list[SitemapNode]:
         """
         Створює Node для кожного URL зі sitemap.
@@ -218,6 +219,7 @@ class SitemapProcessor:
             parent_sitemap_url: URL батьківського sitemap
             depth: Глибина у графі
             max_urls: Максимальна кількість URL для обробки (None = всі)
+            lastmod: ``{url: <lastmod>}`` з того ж sitemap → ``SitemapNode.lastmod``
 
         Returns:
             Список створених SitemapNode
@@ -261,6 +263,7 @@ class SitemapProcessor:
                 url=url,
                 node_type="url",
                 parent_sitemap=parent_sitemap_url,
+                lastmod=(lastmod or {}).get(url),
                 depth=depth,
                 should_scan=False,  # URL вузли не сканують (це кінцеві точки)
                 can_create_edges=False,

{graph_crawler-5.3.4 → graph_crawler-5.3.5}/graph_crawler/application/use_cases/crawling/sitemap_spider.py

@@ -560,6 +560,7 @@ class SitemapSpider(BaseSpider):
                         parent_sitemap_url=sitemap_url,
                         depth=depth + 1,
                         max_urls=budget,
+                        lastmod=sitemap_data.get("lastmod"),
                     )
                     self.urls_extracted += len(url_nodes)
                     if self.max_urls and self.urls_extracted >= self.max_urls:

{graph_crawler-5.3.4 → graph_crawler-5.3.5}/graph_crawler/application/use_cases/crawling/spider.py

@@ -184,6 +184,9 @@ class GraphSpider(BaseSpider):
             self.domain_filter = DomainFilter(domain_config, self.event_bus)
         else:
             self.domain_filter = domain_filter
+        alias_hook = getattr(self.config, "redirect_host_alias", None)
+        if alias_hook is not None and hasattr(self.domain_filter, "redirect_host_alias"):
+            self.domain_filter.redirect_host_alias = alias_hook
 
         # DI: Path filter
         if path_filter is None:

{graph_crawler-5.3.4 → graph_crawler-5.3.5}/graph_crawler/domain/entities/graph.py

@@ -392,20 +392,95 @@ class Graph:
             return self._backend.iter_nodes_sync()  # type: ignore
         return iter(self._nodes.values())
 
-    async def iter_nodes_async(self, batch_size: int = 1000) -> AsyncIterator[Node]:
+    async def iter_nodes_async(
+        self, batch_size: int = 1000, *, include_evicted: bool = False
+    ) -> AsyncIterator[Node]:
         """Streaming-ітератор по нодах. Без завантаження всього графа в RAM.
 
+        Args:
+            batch_size: розмір батчу для backend-ітерації.
+            include_evicted: якщо ``True`` і граф працює без backend у
+                LOW-MEMORY режимі — після RAM-нод доітеровуються й ноди,
+                витіснені на диск (eviction storage). Без цього прапорця
+                merge браузер-рескану бачив лише RAM і губив витіснені
+                seed-вакансії (P23). При ``_use_backend`` evicted-ноди вже
+                в backend, тож прапорець нічого не змінює.
+
         Приклад:
-            async for node in graph.iter_nodes_async():
+            async for node in graph.iter_nodes_async(include_evicted=True):
                 await process(node)
         """
         if self._use_backend and hasattr(self._backend, "iter_nodes"):
             async for node in self._backend.iter_nodes(batch_size=batch_size):  # type: ignore
                 yield node
-        else:
-            for node in self._nodes.values():
+            return
+
+        seen_ids: set[str] = set()
+        for node in list(self._nodes.values()):
+            seen_ids.add(node.node_id)
+            yield node
+
+        if not include_evicted or not self._eviction_storage or not self._evicted_urls:
+            return
+        for url in list(self._evicted_urls):
+            try:
+                node_data = self._eviction_storage.load_node_sync(url)
+            except Exception as exc:  # noqa: BLE001
+                logger.warning("iter_nodes_async: failed to load evicted node %s: %s", url, exc)
+                continue
+            node = self._materialize_evicted_node(node_data)
+            if node is not None and node.node_id not in seen_ids:
+                seen_ids.add(node.node_id)
                 yield node
 
+    def _materialize_evicted_node(self, node_data: Any) -> Node | None:
+        """Будує :class:`Node` з вивантаженого payload БЕЗ повернення в RAM.
+
+        Дзеркалить реконструкцію `_load_node_from_disk` (підклас Node, кастомні
+        поля з `user_data['_custom_fields']`), але не мутує `_nodes`/`_evicted_urls`:
+        ітерація по витіснених нодах має лишати їх на диску (інакше merge
+        матеріалізував би весь граф у RAM — те саме LOW-MEMORY-переповнення).
+        """
+        if node_data is None:
+            return None
+        if isinstance(node_data, Node):
+            return node_data
+        if not isinstance(node_data, dict):
+            return None
+        try:
+            user_data = node_data.get("user_data") or {}
+            if isinstance(user_data, dict) and "_node_class" in user_data:
+                user_data = {k: v for k, v in user_data.items() if k != "_node_class"}
+            raw_custom_fields = (
+                user_data.get("_custom_fields", {}) if isinstance(user_data, dict) else {}
+            )
+            from graph_crawler.application.dto.mappers.node_data_codec import NodeDataCodec
+
+            custom_fields = NodeDataCodec.decode(raw_custom_fields) if raw_custom_fields else {}
+            node_cls: type[Node] = getattr(self, "_node_class", None) or Node
+            node_kwargs = {
+                "url": node_data["url"],
+                "node_id": node_data["node_id"],
+                "depth": node_data["depth"],
+                "scanned": node_data["scanned"],
+            }
+            if custom_fields:
+                node_kwargs.update(
+                    {k: v for k, v in custom_fields.items() if k in node_cls.model_fields}
+                )
+            node = node_cls(**node_kwargs)
+            for attr in ("response_status", "content_hash", "simhash", "priority"):
+                if node_data.get(attr):
+                    setattr(node, attr, node_data[attr])
+            if node_data.get("metadata") is not None:
+                node.metadata = node_data["metadata"]
+            if user_data:
+                node.user_data = user_data
+            return node
+        except Exception as exc:  # noqa: BLE001
+            logger.warning("iter_nodes_async: cannot build Node from evicted payload: %s", exc)
+            return None
+
     def iter_edges(self) -> Iterator[Edge]:
         """Ітератор по всіх ребрах графу."""
         if self._use_backend and hasattr(self._backend, "iter_edges_sync"):
@@ -1986,10 +2061,16 @@ class Graph:
         from itertools import chain
 
         protected = exclude_node_ids or ()
-        # Lazy generators - не створюють проміжні списки
-        scanned = (n for n in self.iter_nodes() if n.scanned and n.node_id not in protected)
+        # Lazy generators - не створюють проміжні списки.
+        # P23: закріплені (pinned) ноди не витісняються — це seed-вакансії
+        # браузер-рескану, які в LOW-MEMORY режимі найстаріші й пропадали першими.
+        scanned = (
+            n for n in self.iter_nodes()
+            if n.scanned and n.node_id not in protected and not getattr(n, "pinned", False)
+        )
         unscanned = (
-            n for n in self.iter_nodes() if not n.scanned and n.node_id not in protected
+            n for n in self.iter_nodes()
+            if not n.scanned and n.node_id not in protected and not getattr(n, "pinned", False)
         )
 
         # chain - lazy concatenation, list() матеріалізує тільки раз
@@ -2352,6 +2433,47 @@ class Graph:
         """
         return len(self._evicted_urls)
 
+    @property
+    def eviction_stats(self) -> dict[str, Any]:
+        """Лічильники eviction самого графа (RAM-mutations), не backend-сховища.
+
+        P23: merge браузер-рескану має читати саме це (граф без backend сам
+        витісняє в LMDB/SQLite, а `backend.get_stats()` цього не бачить і показує
+        0). Копія, щоб споживач не мутував внутрішній стан.
+        """
+        return dict(self._eviction_stats)
+
+    def pin_node(self, node_id_or_url: str) -> bool:
+        """Закріплює ноду (P23): eviction її більше не витісняє.
+
+        Приймає node_id або URL. Для seed-вакансій браузер-рескану це гарантує,
+        що вони доживуть до merge навіть у LOW-MEMORY режимі.
+
+        Returns:
+            True, якщо ноду знайдено в RAM і закріплено.
+        """
+        node = self._nodes.get(node_id_or_url)
+        if node is None:
+            node = self._url_to_node.get(self._normalize_url(node_id_or_url))
+        if node is None:
+            return False
+        node.pinned = True
+        return True
+
+    def pin_seed_nodes(self) -> int:
+        """Закріплює всі seed-ноди (``depth == 0``) у RAM. Повертає їх кількість.
+
+        Seed-и браузер-рескану — це знайдені вакансії; вони найстаріші й у
+        LOW-MEMORY режимі витісняються першими (P23). Викликається перед
+        фазою, що може тригерити eviction.
+        """
+        pinned = 0
+        for node in self._nodes.values():
+            if node.depth == 0 and not node.pinned:
+                node.pinned = True
+                pinned += 1
+        return pinned
+
     def load_all_from_disk(self, *, include_edges: bool = True) -> int:
         """
         Завантажує всі evicted ноди (і, за замовчуванням, ребра) назад в RAM.