web-scanner 2.1.0__tar.gz → 2.1.2__tar.gz

{web_scanner-2.1.0 → web_scanner-2.1.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: web-scanner
-Version: 2.1.0
+Version: 2.1.2
 Summary: An async Textual TUI for website reconnaissance — DNS, WHOIS, TLS, security, tech-stack and SEO, with no paid APIs.
 Project-URL: Homepage, https://github.com/iamramizk/web-scanner
 Project-URL: Repository, https://github.com/iamramizk/web-scanner
@@ -62,10 +62,11 @@ borders are embedded (Natural Earth), so even the map needs no tile service.
 
 ## Features
 
-Nine tabs, scanned concurrently and rendered live as each module finishes:
+Ten tabs, scanned concurrently and rendered live as each module finishes:
 
 - **DNS** — A / AAAA / NS / CNAME / SOA / MX / TXT / CAA / DS / DNSKEY records, plus email
-  authentication folded in: DMARC (`_dmarc`) and DKIM (probes ~40 common selectors).
+  authentication folded in: DMARC (`_dmarc`) and DKIM (probes ~40 common selectors, each
+  labelled with its email provider).
 - **Whois** — parsed system `whois`, with rich gTLD and ccTLD field support (registrar,
   dates, nameservers, per-contact details).
 - **Subdomains** — discovered natively from TLS certificate SANs and `socket` probes of
@@ -76,10 +77,12 @@ Nine tabs, scanned concurrently and rendered live as each module finishes:
   X-Frame-Options, …), and blocklist status across public filtering resolvers
   (AdGuard, CleanBrowsing, Cloudflare, Google, OpenDNS, Quad9).
 - **Headers** — the full set of HTTP response headers.
-- **Tech** — technology-stack detection via [Wappalyzer](https://github.com/tunetheweb/wappalyzer):
-  name, category, confidence, groups and version.
+- **Tech** — technology-stack detection via [Wappalyzer](https://github.com/tunetheweb/wappalyzer),
+  split into one table per technology group, each showing name, category, confidence and version.
 - **SEO** — page content (title/description with length hints, H1–H3, social links), top
   keyword n-grams, `robots.txt` and sitemaps, and JSON-LD structured data.
+- **Sitemap** — the site's URLs discovered from its `sitemap.xml` (recursing into nested
+  sitemap indexes) and rebuilt into a clickable URL-path tree — no crawling.
 - **Links** — internal and external links, with their anchor text.
 
 Alongside the tabs, two fixed panels:
@@ -87,7 +90,7 @@ Alongside the tabs, two fixed panels:
 - **Country map** — real country outlines auto-framed around the server's location, drawn
   with braille characters (`+` / `-` to zoom).
 - **Server** — online status and response time, IP, geolocation, ISP, AS, hosting provider
-  and detected tech.
+  and detected CMS (name and version, from the Tech scan).
 
 ## Installation
 
@@ -142,10 +145,12 @@ pipx install --force git+https://github.com/iamramizk/web-scanner
 
 ### Keys
 
-| Key             | Action                                                              |
-| --------------- | ------------------------------------------------------------------- |
-| `←` / `→` `Tab` | Switch tabs                                                         |
-| `r`             | Rescan                                                              |
-| `s`             | Save — export every tab to CSV under `output/<domain>_<timestamp>/` |
-| `esc`           | Edit the domain and scan a new one                                  |
-| `q`             | Quit                                                                |
+| Key              | Action                                                              |
+| ---------------- | ------------------------------------------------------------------- |
+| `←` / `→` `Tab`  | Switch tabs                                                         |
+| `+` / `-`        | Zoom the country map in / out                                       |
+| `↑` / `↓` `enter`| Navigate the Sitemap tree (`space` expands/collapses all)          |
+| `r`              | Rescan                                                              |
+| `s`              | Save — export every tab to CSV under `output/<domain>_<timestamp>/` |
+| `esc`            | Edit the domain and scan a new one                                  |
+| `q`              | Quit                                                                |

{web_scanner-2.1.0 → web_scanner-2.1.2}/README.md

@@ -14,10 +14,11 @@ borders are embedded (Natural Earth), so even the map needs no tile service.
 
 ## Features
 
-Nine tabs, scanned concurrently and rendered live as each module finishes:
+Ten tabs, scanned concurrently and rendered live as each module finishes:
 
 - **DNS** — A / AAAA / NS / CNAME / SOA / MX / TXT / CAA / DS / DNSKEY records, plus email
-  authentication folded in: DMARC (`_dmarc`) and DKIM (probes ~40 common selectors).
+  authentication folded in: DMARC (`_dmarc`) and DKIM (probes ~40 common selectors, each
+  labelled with its email provider).
 - **Whois** — parsed system `whois`, with rich gTLD and ccTLD field support (registrar,
   dates, nameservers, per-contact details).
 - **Subdomains** — discovered natively from TLS certificate SANs and `socket` probes of
@@ -28,10 +29,12 @@ Nine tabs, scanned concurrently and rendered live as each module finishes:
   X-Frame-Options, …), and blocklist status across public filtering resolvers
   (AdGuard, CleanBrowsing, Cloudflare, Google, OpenDNS, Quad9).
 - **Headers** — the full set of HTTP response headers.
-- **Tech** — technology-stack detection via [Wappalyzer](https://github.com/tunetheweb/wappalyzer):
-  name, category, confidence, groups and version.
+- **Tech** — technology-stack detection via [Wappalyzer](https://github.com/tunetheweb/wappalyzer),
+  split into one table per technology group, each showing name, category, confidence and version.
 - **SEO** — page content (title/description with length hints, H1–H3, social links), top
   keyword n-grams, `robots.txt` and sitemaps, and JSON-LD structured data.
+- **Sitemap** — the site's URLs discovered from its `sitemap.xml` (recursing into nested
+  sitemap indexes) and rebuilt into a clickable URL-path tree — no crawling.
 - **Links** — internal and external links, with their anchor text.
 
 Alongside the tabs, two fixed panels:
@@ -39,7 +42,7 @@ Alongside the tabs, two fixed panels:
 - **Country map** — real country outlines auto-framed around the server's location, drawn
   with braille characters (`+` / `-` to zoom).
 - **Server** — online status and response time, IP, geolocation, ISP, AS, hosting provider
-  and detected tech.
+  and detected CMS (name and version, from the Tech scan).
 
 ## Installation
 
@@ -94,10 +97,12 @@ pipx install --force git+https://github.com/iamramizk/web-scanner
 
 ### Keys
 
-| Key             | Action                                                              |
-| --------------- | ------------------------------------------------------------------- |
-| `←` / `→` `Tab` | Switch tabs                                                         |
-| `r`             | Rescan                                                              |
-| `s`             | Save — export every tab to CSV under `output/<domain>_<timestamp>/` |
-| `esc`           | Edit the domain and scan a new one                                  |
-| `q`             | Quit                                                                |
+| Key              | Action                                                              |
+| ---------------- | ------------------------------------------------------------------- |
+| `←` / `→` `Tab`  | Switch tabs                                                         |
+| `+` / `-`        | Zoom the country map in / out                                       |
+| `↑` / `↓` `enter`| Navigate the Sitemap tree (`space` expands/collapses all)          |
+| `r`              | Rescan                                                              |
+| `s`              | Save — export every tab to CSV under `output/<domain>_<timestamp>/` |
+| `esc`            | Edit the domain and scan a new one                                  |
+| `q`              | Quit                                                                |

{web_scanner-2.1.0 → web_scanner-2.1.2}/webscanner/__init__.py

@@ -1,3 +1,3 @@
 """web-scanner v2 — async website reconnaissance TUI package."""
 
-__version__ = "2.1.0"
+__version__ = "2.1.2"

web_scanner-2.1.2/webscanner/colors.py

@@ -0,0 +1,7 @@
+"""Shared semantic colours used in cell markup across modules and the UI."""
+
+GREEN = "green"
+RED = "#FF6C64"
+#: muted value/secondary text — matches the unselected-tab colour
+#: ($text-muted = white @ 60% over the dark theme bg).
+MUTED = "#9D9D9D"

{web_scanner-2.1.0 → web_scanner-2.1.2}/webscanner/core/models.py

@@ -2,7 +2,7 @@
 
 from __future__ import annotations
 
-from dataclasses import dataclass
+from dataclasses import dataclass, field
 from enum import Enum
 from typing import Any
 
@@ -62,16 +62,39 @@ class Sections(list):
     """
 
 
+@dataclass(slots=True)
+class TreeNode:
+    """One node in a hierarchical result (e.g. the Sitemap tab's URL tree).
+
+    The module returns a synthetic root ``TreeNode`` whose descendants mirror the
+    site's URL-path hierarchy (``/blog`` → ``/blog/post-1`` …). A node is a branch
+    (expandable folder) iff it has ``children``; otherwise it's a leaf (a page). The
+    UI turns the root into a Textual ``Tree`` widget. ``total`` (set on the root only)
+    is the count of page URLs behind the tree, shown in the panel subtitle. ``url`` is
+    the full page URL a node stands for (set on leaves), so the UI can make it clickable.
+    """
+
+    label: str
+    children: list["TreeNode"] = field(default_factory=list)
+    total: int | None = None
+    url: str | None = None
+
+
 class Grid(list):
     """A multi-column table result: rows (list of value-lists) + column headers.
 
     A ``list`` of rows, so an empty ``Grid`` reports EMPTY via ``_is_empty``. The
-    first column is treated as each row's primary name.
+    first column is treated as each row's primary name. ``widths`` optionally pins
+    each column to a fixed character width (else content-fit) — used when several
+    Grids share a tab (e.g. Tech's per-group tables) so they all line up.
     """
 
-    def __init__(self, columns: list[str], rows: Any) -> None:
+    def __init__(
+        self, columns: list[str], rows: Any, widths: list[int] | None = None
+    ) -> None:
         super().__init__(rows)
         self.columns = list(columns)
+        self.widths = list(widths) if widths else None
 
     @property
     def names(self) -> list[str]:

{web_scanner-2.1.0 → web_scanner-2.1.2}/webscanner/core/scanner.py

@@ -31,7 +31,7 @@ def _is_empty(data: object) -> bool:
         return True
     if isinstance(data, (dict, list, str, tuple, set)):
         return len(data) == 0
-    return False
+    return False  # a returned TreeNode is DONE; modules signal empty by returning None
 
 
 class AsyncScanner:

{web_scanner-2.1.0 → web_scanner-2.1.2}/webscanner/modules/__init__.py

@@ -1,9 +1,10 @@
 """Scan module registry.
 
 ``all_modules()`` order defines the tab order in the UI:
-DNS · Whois · Subdomains · SSL · Security · Headers · Tech · SEO · Links
+DNS · Whois · Subdomains · SSL · Security · Headers · Tech · SEO · Sitemap · Links
 (Email auth — DMARC/DKIM — is folded into DNS. Security = ports + HTTP headers + blocklists.
-SEO = schema + content + keywords. Links = internal + external page links.)
+SEO = schema + content + keywords. Sitemap = the site's published sitemap(s) as a tree.
+Links = internal + external page links.)
 """
 
 from __future__ import annotations
@@ -14,6 +15,7 @@ from .headers import HeadersModule
 from .links import LinksModule
 from .security import SecurityModule
 from .seo import SeoModule
+from .sitemap import SitemapModule
 from .ssl import SslModule
 from .subdomains import SubdomainsModule
 from .tech import TechModule
@@ -31,6 +33,7 @@ def all_modules() -> list[ScanModule]:
         HeadersModule(),
         TechModule(),
         SeoModule(),
+        SitemapModule(),
         LinksModule(),
     ]
 

{web_scanner-2.1.0 → web_scanner-2.1.2}/webscanner/modules/dns.py

@@ -19,40 +19,44 @@ from ..core.context import ScanContext
 RECORD_TYPES = ("A", "AAAA", "NS", "CNAME", "SOA", "MX", "TXT", "CAA", "DS", "DNSKEY")
 
 # Known DKIM selectors across the major providers/ESPs. DKIM can't be enumerated
-# from DNS alone, so we probe this list of well-known selectors.
-DKIM_SELECTORS = (
-    # generic
-    "default", "dkim", "mail", "selector", "s", "s1", "s2", "key1", "dk",
+# from DNS alone, so we probe this list of well-known selectors. The value is the
+# provider a hit points to (shown next to the selector); an empty string means the
+# selector is generic and doesn't identify a provider on its own.
+DKIM_SELECTORS: dict[str, str] = {
+    # generic — the selector name alone doesn't identify a provider
+    "default": "", "dkim": "", "mail": "", "selector": "", "s": "",
+    "s1": "", "s2": "", "key1": "", "dk": "",
     # Microsoft 365 / Outlook
-    "selector1", "selector2",
+    "selector1": "Microsoft 365", "selector2": "Microsoft 365",
     # Google Workspace
-    "google",
+    "google": "Google Workspace",
     # Amazon SES
-    "amazonses",
+    "amazonses": "Amazon SES",
     # Zoho
-    "zoho", "zmail",
+    "zoho": "Zoho", "zmail": "Zoho",
     # ProtonMail
-    "protonmail", "protonmail2", "protonmail3",
+    "protonmail": "Proton Mail", "protonmail2": "Proton Mail", "protonmail3": "Proton Mail",
     # Fastmail
-    "fm1", "fm2", "fm3",
+    "fm1": "Fastmail", "fm2": "Fastmail", "fm3": "Fastmail",
     # Mailchimp / Mandrill
-    "k1", "k2", "k3", "mandrill",
+    "k1": "Mailchimp", "k2": "Mailchimp", "k3": "Mailchimp", "mandrill": "Mandrill",
     # SendGrid
-    "smtpapi",
+    "smtpapi": "SendGrid",
     # Postmark
-    "pm",
+    "pm": "Postmark",
     # Mailgun / generic smtp
-    "smtp", "mg", "mailo",
+    "smtp": "", "mg": "Mailgun", "mailo": "",
     # Campaign Monitor
-    "cm",
+    "cm": "Campaign Monitor",
     # Apple iCloud
-    "sig1",
+    "sig1": "Apple iCloud",
     # HubSpot
-    "hs1", "hs2",
+    "hs1": "HubSpot", "hs2": "HubSpot",
     # misc common
-    "mxvault", "everlytickey1", "everlytickey2", "titan1", "titan2",
-    "turbo-smtp", "mailjet", "sendinblue", "klaviyo",
-)
+    "mxvault": "MXVault", "everlytickey1": "Everlytic", "everlytickey2": "Everlytic",
+    "titan1": "Titan", "titan2": "Titan", "turbo-smtp": "TurboSMTP",
+    "mailjet": "Mailjet", "sendinblue": "Brevo (Sendinblue)", "klaviyo": "Klaviyo",
+}
 
 
 def _is_dkim(records: list[str]) -> bool:
@@ -92,5 +96,8 @@ class DnsModule(ScanModule):
             out["DMARC"] = dmarc_records
         found = [s for s in dkim_hits if s]
         if found:
-            out["DKIM"] = found
+            out["DKIM"] = [
+                f"{s}  ({DKIM_SELECTORS[s]})" if DKIM_SELECTORS[s] else s
+                for s in found
+            ]
         return out

{web_scanner-2.1.0 → web_scanner-2.1.2}/webscanner/modules/security.py

@@ -11,7 +11,7 @@ from __future__ import annotations
 
 import asyncio
 
-from ..colors import GREEN, RED
+from ..colors import GREEN, RED, MUTED
 from ..core.module import ScanModule
 from ..core.context import ScanContext
 from ..core.models import Section, Sections
@@ -71,7 +71,7 @@ async def _port_open(ip: str, port: int) -> bool:
 async def _scan_ports(ip: str) -> dict[str, str]:
     opens = await asyncio.gather(*(_port_open(ip, p) for p in COMMON_PORTS))
     return {
-        f"{port} ({svc})": f"[{GREEN}]open[/]" if is_open else "[dim]closed[/]"
+        f"{port} ({svc})": f"[{GREEN}]open[/]" if is_open else f"[{MUTED}]closed[/]"
         for (port, svc), is_open in zip(COMMON_PORTS.items(), opens)
     }
 

{web_scanner-2.1.0 → web_scanner-2.1.2}/webscanner/modules/seo.py

@@ -22,7 +22,7 @@ import requests
 from bs4 import BeautifulSoup
 from rich.markup import escape
 
-from ..colors import GREEN, RED
+from ..colors import GREEN, RED, MUTED
 from ..core.module import ScanModule
 from ..core.context import ScanContext
 from ..core.models import Section, Sections
@@ -46,10 +46,56 @@ _STOPWORDS = frozenset(
 )
 
 
+# JSON syntax-highlight colours (dark-friendly, aligned with the app palette).
+_J_KEY = "#5FAFFF"   # object keys — blue
+_J_STR = GREEN       # string values
+_J_NUM = "#D7AF87"   # numbers — amber
+_J_KW = RED          # true / false / null
+_J_PUNCT = MUTED     # braces, brackets, commas, colons
+
+_JSON_TOKEN = re.compile(
+    r'(?P<space>\s+)'
+    r'|(?P<str>"(?:[^"\\]|\\.)*")'
+    r'|(?P<num>-?\d+(?:\.\d+)?(?:[eE][+-]?\d+)?)'
+    r'|(?P<kw>true|false|null)'
+    r'|(?P<punct>[{}\[\],:])'
+)
+
+
+def _highlight_json(obj: object) -> str:
+    """Pretty-print `obj` as indented JSON with Rich colour markup. Brackets in
+    the payload are escaped so Rich doesn't parse them; `_value_cell` renders the
+    markup and CSV export strips it back to plain JSON."""
+    raw = json.dumps(obj, indent=2, ensure_ascii=False)
+    tokens = list(_JSON_TOKEN.finditer(raw))
+    out: list[str] = []
+    pos = 0
+    for i, m in enumerate(tokens):
+        if m.start() > pos:  # any char the tokenizer skipped, kept verbatim
+            out.append(escape(raw[pos:m.start()]))
+        pos = m.end()
+        kind, text = m.lastgroup, m.group()
+        if kind == "space":
+            out.append(text)
+        elif kind == "str":
+            # a string is a key iff the next non-space token is a colon
+            nxt = next((t for t in tokens[i + 1:] if t.lastgroup != "space"), None)
+            colour = _J_KEY if nxt and nxt.group() == ":" else _J_STR
+            out.append(f"[{colour}]{escape(text)}[/]")
+        elif kind == "num":
+            out.append(f"[{_J_NUM}]{text}[/]")
+        elif kind == "kw":
+            out.append(f"[{_J_KW}]{text}[/]")
+        else:  # punct
+            out.append(f"[{_J_PUNCT}]{escape(text)}[/]")
+    out.append(escape(raw[pos:]))
+    return "".join(out)
+
+
 def _len_line(text: str, lo: int, hi: int) -> str:
     n = len(text)
     colour = GREEN if lo <= n <= hi else RED
-    return f"[dim]{escape(text)}[/]\n[{colour}]{n} chars · rec. {lo}-{hi}[/]"
+    return f"[{MUTED}]{escape(text)}[/]\n[{colour}]{n} chars · rec. {lo}-{hi}[/]"
 
 
 class SeoModule(ScanModule):
@@ -88,8 +134,8 @@ class SeoModule(ScanModule):
                 pass
         schema: dict[str, str] = {"Has Schema": f"[{GREEN}]Yes[/]" if blocks else f"[{RED}]No[/]"}
         if blocks:
-            schema["Schema"] = json.dumps(
-                blocks[0] if len(blocks) == 1 else blocks, indent=2, ensure_ascii=False
+            schema["Schema"] = _highlight_json(
+                blocks[0] if len(blocks) == 1 else blocks
             )
 
         # --- Content ---

web_scanner-2.1.2/webscanner/modules/sitemap.py

@@ -0,0 +1,202 @@
+"""Sitemap module — the site's URLs as a folder tree built from their paths.
+
+This does **not** crawl the site: it discovers the sitemap(s) the site already
+publishes (``Sitemap:`` lines in robots.txt, falling back to ``/sitemap.xml`` and
+``/sitemap_index.xml``), fetches them (recursing into any nested ``<sitemapindex>``
+files, gunzipping ``.xml.gz``), and collects every page ``<loc>``.
+
+Those flat URLs are then rebuilt into a hierarchy keyed by their **path segments**,
+so the tree reads like the site's folder structure — first level is the first slug
+(``/blog``, ``/news``, ``/wp-content`` …), leaves are the final pages — rather than
+mirroring the sitemap *files*. If the sitemaps span more than one host (e.g.
+subdomains), the differing host becomes the top level so URLs aren't merged.
+
+Robustness: a ``visited`` set + caps on total sitemap files, collected URLs and
+sitemap-index nesting depth guard against loops and pathologically huge sitemaps.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import gzip
+import xml.etree.ElementTree as ET
+from urllib.parse import urlparse
+
+import requests
+
+from ..core.module import ScanModule
+from ..core.context import ScanContext
+from ..core.models import TreeNode
+from ..net.http import DEFAULT_HEADERS, TIMEOUT
+
+#: total sitemap files fetched across the whole tree (loop / fan-out guard)
+MAX_SITEMAPS = 60
+#: total page URLs collected before we stop (bounds the tree size)
+MAX_URLS = 10000
+#: how deep a chain of nested <sitemapindex> files we follow
+MAX_DEPTH = 6
+
+
+class SitemapModule(ScanModule):
+    name = "sitemap"
+    label = "Sitemap"
+
+    async def run(self, ctx: ScanContext) -> TreeNode | None:
+        return await asyncio.to_thread(self._build, ctx)
+
+    # ---- build (blocking; runs in a thread) -------------------------------
+
+    def _build(self, ctx: ScanContext) -> TreeNode | None:
+        urls = self._collect(ctx.domain)
+        if not urls:
+            return None  # -> EMPTY
+        truncated = len(urls) >= MAX_URLS
+        return _url_tree(urls, truncated)
+
+    def _collect(self, domain: str) -> list[str]:
+        """Fetch every sitemap (recursing indexes) and return the page URLs found."""
+        visited: set[str] = set()
+        urls: list[str] = []
+        budget = [MAX_SITEMAPS]  # mutable counter shared across the recursion
+        for sitemap in self._discover(domain):
+            self._crawl(sitemap, visited, urls, budget, depth=0)
+            if len(urls) >= MAX_URLS:
+                break
+        seen: set[str] = set()  # dedupe, preserve order
+        return [u for u in urls if not (u in seen or seen.add(u))]
+
+    def _crawl(self, url: str, visited: set[str], urls: list[str], budget: list[int], depth: int) -> None:
+        if url in visited or budget[0] <= 0 or depth > MAX_DEPTH or len(urls) >= MAX_URLS:
+            return
+        visited.add(url)
+        budget[0] -= 1
+
+        raw = self._get(url)
+        if raw is None:
+            return
+        kind, locs = _parse_sitemap(raw)
+        if kind == "index":
+            for loc in locs:
+                self._crawl(loc, visited, urls, budget, depth + 1)
+                if len(urls) >= MAX_URLS:
+                    return
+        elif kind == "urlset":
+            for loc in locs:
+                urls.append(loc)
+                if len(urls) >= MAX_URLS:
+                    return
+
+    # ---- discovery + fetch ------------------------------------------------
+
+    def _discover(self, domain: str) -> list[str]:
+        """Sitemap URLs from robots.txt, else the two conventional defaults."""
+        urls = self._robots_sitemaps(domain)
+        if not urls:
+            base = f"https://{domain}"
+            urls = [f"{base}/sitemap.xml", f"{base}/sitemap_index.xml"]
+        seen: set[str] = set()
+        return [u for u in urls if not (u in seen or seen.add(u))]
+
+    @staticmethod
+    def _robots_sitemaps(domain: str) -> list[str]:
+        try:
+            resp = requests.get(
+                f"https://{domain}/robots.txt",
+                headers=DEFAULT_HEADERS, timeout=TIMEOUT, allow_redirects=True,
+            )
+            if resp.status_code != 200:
+                return []
+            return [
+                ln.split(":", 1)[1].strip()
+                for ln in resp.text.splitlines()
+                if ln.strip().lower().startswith("sitemap:")
+            ]
+        except Exception:  # noqa: BLE001
+            return []
+
+    @staticmethod
+    def _get(url: str) -> bytes | None:
+        """Fetch a sitemap; gunzip ``.xml.gz`` / gzip payloads by magic bytes."""
+        try:
+            resp = requests.get(url, headers=DEFAULT_HEADERS, timeout=TIMEOUT, allow_redirects=True)
+            if resp.status_code != 200:
+                return None
+            raw = resp.content
+            if raw[:2] == b"\x1f\x8b":  # gzip magic — a served .gz file (not transport enc.)
+                raw = gzip.decompress(raw)
+            return raw
+        except Exception:  # noqa: BLE001
+            return None
+
+
+# ---- parsing + tree building ----------------------------------------------
+
+
+def _parse_sitemap(raw: bytes) -> tuple[str | None, list[str]]:
+    """Return ``("index"|"urlset", [loc, …])`` or ``(None, [])`` if unparseable."""
+    try:
+        root = ET.fromstring(raw)
+    except ET.ParseError:
+        return None, []
+    locs = [
+        el.text.strip()
+        for el in root.iter()
+        if _localname(el.tag) == "loc" and el.text and el.text.strip()
+    ]
+    kind = "index" if _localname(root.tag) == "sitemapindex" else "urlset"
+    return kind, locs
+
+
+def _localname(tag: str) -> str:
+    """Strip the ``{namespace}`` prefix ElementTree keeps on namespaced tags."""
+    return tag.rsplit("}", 1)[-1].lower()
+
+
+def _segments(url: str, include_host: bool) -> list[str]:
+    """Path segments for `url` as tree labels: ``/blog``, ``/post`` … An empty path
+    (the homepage) yields ``[]`` — it *is* the ``/`` root, not a child of it. When
+    `include_host`, a leading host label (no slash) groups multi-host sitemaps. Any
+    query is kept on the final leaf."""
+    p = urlparse(url)
+    parts = [s for s in p.path.split("/") if s]
+    segs: list[str] = []
+    if include_host and p.netloc:
+        segs.append(p.netloc)
+    segs += ["/" + s for s in parts]
+    if p.query and segs:
+        segs[-1] = segs[-1] + "?" + p.query
+    return segs
+
+
+def _url_tree(urls: list[str], truncated: bool) -> TreeNode:
+    """Build a path-keyed tree under a visible ``/`` root. Within every level,
+    branches (toggleable folders) sort first, then leaf pages — both alphabetical."""
+    include_host = len({urlparse(u).netloc for u in urls if urlparse(u).netloc}) > 1
+    root = TreeNode(label="/")  # the site root / homepage; shown + expanded by the UI
+    index: dict[tuple[str, ...], TreeNode] = {(): root}
+    for url in sorted(urls):
+        parent = root
+        node = root
+        key: tuple[str, ...] = ()
+        for seg in _segments(url, include_host):
+            key += (seg,)
+            node = index.get(key)
+            if node is None:
+                node = TreeNode(label=seg)
+                index[key] = node
+                parent.children.append(node)
+            parent = node
+        node.url = url  # deepest node for this URL (a leaf, unless it later gains children)
+    if truncated:
+        root.children.append(TreeNode(label=f"… (truncated at {MAX_URLS} URLs)"))
+    root.total = len(urls)
+    _sort_tree(root)
+    return root
+
+
+def _sort_tree(node: TreeNode) -> None:
+    """Recursively order children: branches (have children) before leaves, each
+    group alphabetical by label."""
+    for child in node.children:
+        _sort_tree(child)
+    node.children.sort(key=lambda c: (not c.children, c.label.lower()))