web-scanner 2.1.1__tar.gz → 2.1.3__tar.gz

{web_scanner-2.1.1 → web_scanner-2.1.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: web-scanner
-Version: 2.1.1
+Version: 2.1.3
 Summary: An async Textual TUI for website reconnaissance — DNS, WHOIS, TLS, security, tech-stack and SEO, with no paid APIs.
 Project-URL: Homepage, https://github.com/iamramizk/web-scanner
 Project-URL: Repository, https://github.com/iamramizk/web-scanner
@@ -62,10 +62,11 @@ borders are embedded (Natural Earth), so even the map needs no tile service.
 
 ## Features
 
-Nine tabs, scanned concurrently and rendered live as each module finishes:
+Ten tabs, scanned concurrently and rendered live as each module finishes:
 
 - **DNS** — A / AAAA / NS / CNAME / SOA / MX / TXT / CAA / DS / DNSKEY records, plus email
-  authentication folded in: DMARC (`_dmarc`) and DKIM (probes ~40 common selectors).
+  authentication folded in: DMARC (`_dmarc`) and DKIM (probes ~40 common selectors, each
+  labelled with its email provider).
 - **Whois** — parsed system `whois`, with rich gTLD and ccTLD field support (registrar,
   dates, nameservers, per-contact details).
 - **Subdomains** — discovered natively from TLS certificate SANs and `socket` probes of
@@ -76,10 +77,12 @@ Nine tabs, scanned concurrently and rendered live as each module finishes:
   X-Frame-Options, …), and blocklist status across public filtering resolvers
   (AdGuard, CleanBrowsing, Cloudflare, Google, OpenDNS, Quad9).
 - **Headers** — the full set of HTTP response headers.
-- **Tech** — technology-stack detection via [Wappalyzer](https://github.com/tunetheweb/wappalyzer):
-  name, category, confidence, groups and version.
+- **Tech** — technology-stack detection via [Wappalyzer](https://github.com/tunetheweb/wappalyzer),
+  split into one table per technology group, each showing name, category, confidence and version.
 - **SEO** — page content (title/description with length hints, H1–H3, social links), top
   keyword n-grams, `robots.txt` and sitemaps, and JSON-LD structured data.
+- **Sitemap** — the site's URLs discovered from its `sitemap.xml` (recursing into nested
+  sitemap indexes) and rebuilt into a clickable URL-path tree — no crawling.
 - **Links** — internal and external links, with their anchor text.
 
 Alongside the tabs, two fixed panels:
@@ -87,7 +90,7 @@ Alongside the tabs, two fixed panels:
 - **Country map** — real country outlines auto-framed around the server's location, drawn
   with braille characters (`+` / `-` to zoom).
 - **Server** — online status and response time, IP, geolocation, ISP, AS, hosting provider
-  and detected tech.
+  and detected CMS (name and version, from the Tech scan).
 
 ## Installation
 
@@ -142,10 +145,12 @@ pipx install --force git+https://github.com/iamramizk/web-scanner
 
 ### Keys
 
-| Key             | Action                                                              |
-| --------------- | ------------------------------------------------------------------- |
-| `←` / `→` `Tab` | Switch tabs                                                         |
-| `r`             | Rescan                                                              |
-| `s`             | Save — export every tab to CSV under `output/<domain>_<timestamp>/` |
-| `esc`           | Edit the domain and scan a new one                                  |
-| `q`             | Quit                                                                |
+| Key              | Action                                                              |
+| ---------------- | ------------------------------------------------------------------- |
+| `←` / `→` `Tab`  | Switch tabs                                                         |
+| `+` / `-`        | Zoom the country map in / out                                       |
+| `↑` / `↓` `enter`| Navigate the Sitemap tree (`space` expands/collapses all)          |
+| `r`              | Rescan                                                              |
+| `s`              | Save — export every tab to CSV under `output/<domain>_<timestamp>/` |
+| `esc`            | Edit the domain and scan a new one                                  |
+| `q`              | Quit                                                                |

{web_scanner-2.1.1 → web_scanner-2.1.3}/README.md

@@ -14,10 +14,11 @@ borders are embedded (Natural Earth), so even the map needs no tile service.
 
 ## Features
 
-Nine tabs, scanned concurrently and rendered live as each module finishes:
+Ten tabs, scanned concurrently and rendered live as each module finishes:
 
 - **DNS** — A / AAAA / NS / CNAME / SOA / MX / TXT / CAA / DS / DNSKEY records, plus email
-  authentication folded in: DMARC (`_dmarc`) and DKIM (probes ~40 common selectors).
+  authentication folded in: DMARC (`_dmarc`) and DKIM (probes ~40 common selectors, each
+  labelled with its email provider).
 - **Whois** — parsed system `whois`, with rich gTLD and ccTLD field support (registrar,
   dates, nameservers, per-contact details).
 - **Subdomains** — discovered natively from TLS certificate SANs and `socket` probes of
@@ -28,10 +29,12 @@ Nine tabs, scanned concurrently and rendered live as each module finishes:
   X-Frame-Options, …), and blocklist status across public filtering resolvers
   (AdGuard, CleanBrowsing, Cloudflare, Google, OpenDNS, Quad9).
 - **Headers** — the full set of HTTP response headers.
-- **Tech** — technology-stack detection via [Wappalyzer](https://github.com/tunetheweb/wappalyzer):
-  name, category, confidence, groups and version.
+- **Tech** — technology-stack detection via [Wappalyzer](https://github.com/tunetheweb/wappalyzer),
+  split into one table per technology group, each showing name, category, confidence and version.
 - **SEO** — page content (title/description with length hints, H1–H3, social links), top
   keyword n-grams, `robots.txt` and sitemaps, and JSON-LD structured data.
+- **Sitemap** — the site's URLs discovered from its `sitemap.xml` (recursing into nested
+  sitemap indexes) and rebuilt into a clickable URL-path tree — no crawling.
 - **Links** — internal and external links, with their anchor text.
 
 Alongside the tabs, two fixed panels:
@@ -39,7 +42,7 @@ Alongside the tabs, two fixed panels:
 - **Country map** — real country outlines auto-framed around the server's location, drawn
   with braille characters (`+` / `-` to zoom).
 - **Server** — online status and response time, IP, geolocation, ISP, AS, hosting provider
-  and detected tech.
+  and detected CMS (name and version, from the Tech scan).
 
 ## Installation
 
@@ -94,10 +97,12 @@ pipx install --force git+https://github.com/iamramizk/web-scanner
 
 ### Keys
 
-| Key             | Action                                                              |
-| --------------- | ------------------------------------------------------------------- |
-| `←` / `→` `Tab` | Switch tabs                                                         |
-| `r`             | Rescan                                                              |
-| `s`             | Save — export every tab to CSV under `output/<domain>_<timestamp>/` |
-| `esc`           | Edit the domain and scan a new one                                  |
-| `q`             | Quit                                                                |
+| Key              | Action                                                              |
+| ---------------- | ------------------------------------------------------------------- |
+| `←` / `→` `Tab`  | Switch tabs                                                         |
+| `+` / `-`        | Zoom the country map in / out                                       |
+| `↑` / `↓` `enter`| Navigate the Sitemap tree (`space` expands/collapses all)          |
+| `r`              | Rescan                                                              |
+| `s`              | Save — export every tab to CSV under `output/<domain>_<timestamp>/` |
+| `esc`            | Edit the domain and scan a new one                                  |
+| `q`              | Quit                                                                |

{web_scanner-2.1.1 → web_scanner-2.1.3}/webscanner/__init__.py

@@ -1,3 +1,3 @@
 """web-scanner v2 — async website reconnaissance TUI package."""
 
-__version__ = "2.1.1"
+__version__ = "2.1.3"

{web_scanner-2.1.1 → web_scanner-2.1.3}/webscanner/colors.py

@@ -5,3 +5,6 @@ RED = "#FF6C64"
 #: muted value/secondary text — matches the unselected-tab colour
 #: ($text-muted = white @ 60% over the dark theme bg).
 MUTED = "#9D9D9D"
+#: activity-log body text — one step down the white → MUTED ladder, so a log line
+#: reads softer than a heading but stays well clear of the dim MUTED timestamp.
+BODY = "#C6C6C6"

{web_scanner-2.1.1 → web_scanner-2.1.3}/webscanner/core/models.py

@@ -2,7 +2,7 @@
 
 from __future__ import annotations
 
-from dataclasses import dataclass
+from dataclasses import dataclass, field
 from enum import Enum
 from typing import Any
 
@@ -62,16 +62,39 @@ class Sections(list):
     """
 
 
+@dataclass(slots=True)
+class TreeNode:
+    """One node in a hierarchical result (e.g. the Sitemap tab's URL tree).
+
+    The module returns a synthetic root ``TreeNode`` whose descendants mirror the
+    site's URL-path hierarchy (``/blog`` → ``/blog/post-1`` …). A node is a branch
+    (expandable folder) iff it has ``children``; otherwise it's a leaf (a page). The
+    UI turns the root into a Textual ``Tree`` widget. ``total`` (set on the root only)
+    is the count of page URLs behind the tree, shown in the panel subtitle. ``url`` is
+    the full page URL a node stands for (set on leaves), so the UI can make it clickable.
+    """
+
+    label: str
+    children: list["TreeNode"] = field(default_factory=list)
+    total: int | None = None
+    url: str | None = None
+
+
 class Grid(list):
     """A multi-column table result: rows (list of value-lists) + column headers.
 
     A ``list`` of rows, so an empty ``Grid`` reports EMPTY via ``_is_empty``. The
-    first column is treated as each row's primary name.
+    first column is treated as each row's primary name. ``widths`` optionally pins
+    each column to a fixed character width (else content-fit) — used when several
+    Grids share a tab (e.g. Tech's per-group tables) so they all line up.
     """
 
-    def __init__(self, columns: list[str], rows: Any) -> None:
+    def __init__(
+        self, columns: list[str], rows: Any, widths: list[int] | None = None
+    ) -> None:
         super().__init__(rows)
         self.columns = list(columns)
+        self.widths = list(widths) if widths else None
 
     @property
     def names(self) -> list[str]:

{web_scanner-2.1.1 → web_scanner-2.1.3}/webscanner/core/scanner.py

@@ -31,7 +31,7 @@ def _is_empty(data: object) -> bool:
         return True
     if isinstance(data, (dict, list, str, tuple, set)):
         return len(data) == 0
-    return False
+    return False  # a returned TreeNode is DONE; modules signal empty by returning None
 
 
 class AsyncScanner:

{web_scanner-2.1.1 → web_scanner-2.1.3}/webscanner/modules/__init__.py

@@ -1,9 +1,10 @@
 """Scan module registry.
 
 ``all_modules()`` order defines the tab order in the UI:
-DNS · Whois · Subdomains · SSL · Security · Headers · Tech · SEO · Links
+DNS · Whois · Subdomains · SSL · Security · Headers · Tech · SEO · Sitemap · Links
 (Email auth — DMARC/DKIM — is folded into DNS. Security = ports + HTTP headers + blocklists.
-SEO = schema + content + keywords. Links = internal + external page links.)
+SEO = schema + content + keywords. Sitemap = the site's published sitemap(s) as a tree.
+Links = internal + external page links.)
 """
 
 from __future__ import annotations
@@ -14,6 +15,7 @@ from .headers import HeadersModule
 from .links import LinksModule
 from .security import SecurityModule
 from .seo import SeoModule
+from .sitemap import SitemapModule
 from .ssl import SslModule
 from .subdomains import SubdomainsModule
 from .tech import TechModule
@@ -31,6 +33,7 @@ def all_modules() -> list[ScanModule]:
         HeadersModule(),
         TechModule(),
         SeoModule(),
+        SitemapModule(),
         LinksModule(),
     ]
 

{web_scanner-2.1.1 → web_scanner-2.1.3}/webscanner/modules/dns.py

@@ -19,40 +19,44 @@ from ..core.context import ScanContext
 RECORD_TYPES = ("A", "AAAA", "NS", "CNAME", "SOA", "MX", "TXT", "CAA", "DS", "DNSKEY")
 
 # Known DKIM selectors across the major providers/ESPs. DKIM can't be enumerated
-# from DNS alone, so we probe this list of well-known selectors.
-DKIM_SELECTORS = (
-    # generic
-    "default", "dkim", "mail", "selector", "s", "s1", "s2", "key1", "dk",
+# from DNS alone, so we probe this list of well-known selectors. The value is the
+# provider a hit points to (shown next to the selector); an empty string means the
+# selector is generic and doesn't identify a provider on its own.
+DKIM_SELECTORS: dict[str, str] = {
+    # generic — the selector name alone doesn't identify a provider
+    "default": "", "dkim": "", "mail": "", "selector": "", "s": "",
+    "s1": "", "s2": "", "key1": "", "dk": "",
     # Microsoft 365 / Outlook
-    "selector1", "selector2",
+    "selector1": "Microsoft 365", "selector2": "Microsoft 365",
     # Google Workspace
-    "google",
+    "google": "Google Workspace",
     # Amazon SES
-    "amazonses",
+    "amazonses": "Amazon SES",
     # Zoho
-    "zoho", "zmail",
+    "zoho": "Zoho", "zmail": "Zoho",
     # ProtonMail
-    "protonmail", "protonmail2", "protonmail3",
+    "protonmail": "Proton Mail", "protonmail2": "Proton Mail", "protonmail3": "Proton Mail",
     # Fastmail
-    "fm1", "fm2", "fm3",
+    "fm1": "Fastmail", "fm2": "Fastmail", "fm3": "Fastmail",
     # Mailchimp / Mandrill
-    "k1", "k2", "k3", "mandrill",
+    "k1": "Mailchimp", "k2": "Mailchimp", "k3": "Mailchimp", "mandrill": "Mandrill",
     # SendGrid
-    "smtpapi",
+    "smtpapi": "SendGrid",
     # Postmark
-    "pm",
+    "pm": "Postmark",
     # Mailgun / generic smtp
-    "smtp", "mg", "mailo",
+    "smtp": "", "mg": "Mailgun", "mailo": "",
     # Campaign Monitor
-    "cm",
+    "cm": "Campaign Monitor",
     # Apple iCloud
-    "sig1",
+    "sig1": "Apple iCloud",
     # HubSpot
-    "hs1", "hs2",
+    "hs1": "HubSpot", "hs2": "HubSpot",
     # misc common
-    "mxvault", "everlytickey1", "everlytickey2", "titan1", "titan2",
-    "turbo-smtp", "mailjet", "sendinblue", "klaviyo",
-)
+    "mxvault": "MXVault", "everlytickey1": "Everlytic", "everlytickey2": "Everlytic",
+    "titan1": "Titan", "titan2": "Titan", "turbo-smtp": "TurboSMTP",
+    "mailjet": "Mailjet", "sendinblue": "Brevo (Sendinblue)", "klaviyo": "Klaviyo",
+}
 
 
 def _is_dkim(records: list[str]) -> bool:
@@ -92,5 +96,8 @@ class DnsModule(ScanModule):
             out["DMARC"] = dmarc_records
         found = [s for s in dkim_hits if s]
         if found:
-            out["DKIM"] = found
+            out["DKIM"] = [
+                f"{s}  ({DKIM_SELECTORS[s]})" if DKIM_SELECTORS[s] else s
+                for s in found
+            ]
         return out

{web_scanner-2.1.1 → web_scanner-2.1.3}/webscanner/modules/links.py

@@ -18,6 +18,11 @@ from ..core.models import Section, Sections
 
 _SKIP_PREFIXES = ("#", "javascript:", "mailto:", "tel:", "data:")
 
+# Placeholder row shown when a section found nothing. Named so callers can tell
+# "no links" from "one link" — the section is never empty, so len() can't.
+EMPTY_INTERNAL = ("—", "no internal links found")
+EMPTY_EXTERNAL = ("—", "no external links found")
+
 
 class LinksModule(ScanModule):
     name = "links"
@@ -31,8 +36,8 @@ class LinksModule(ScanModule):
             internal, external = await asyncio.to_thread(self._parse, ctx)
 
         return Sections([
-            Section("Internal", internal or [("—", "no internal links found")], ("Link text", "URL")),
-            Section("External", external or [("—", "no external links found")], ("Link text", "URL")),
+            Section("Internal", internal or [EMPTY_INTERNAL], ("Link text", "URL")),
+            Section("External", external or [EMPTY_EXTERNAL], ("Link text", "URL")),
         ])
 
     @staticmethod

web_scanner-2.1.3/webscanner/modules/sitemap.py

@@ -0,0 +1,202 @@
+"""Sitemap module — the site's URLs as a folder tree built from their paths.
+
+This does **not** crawl the site: it discovers the sitemap(s) the site already
+publishes (``Sitemap:`` lines in robots.txt, falling back to ``/sitemap.xml`` and
+``/sitemap_index.xml``), fetches them (recursing into any nested ``<sitemapindex>``
+files, gunzipping ``.xml.gz``), and collects every page ``<loc>``.
+
+Those flat URLs are then rebuilt into a hierarchy keyed by their **path segments**,
+so the tree reads like the site's folder structure — first level is the first slug
+(``/blog``, ``/news``, ``/wp-content`` …), leaves are the final pages — rather than
+mirroring the sitemap *files*. If the sitemaps span more than one host (e.g.
+subdomains), the differing host becomes the top level so URLs aren't merged.
+
+Robustness: a ``visited`` set + caps on total sitemap files, collected URLs and
+sitemap-index nesting depth guard against loops and pathologically huge sitemaps.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import gzip
+import xml.etree.ElementTree as ET
+from urllib.parse import urlparse
+
+import requests
+
+from ..core.module import ScanModule
+from ..core.context import ScanContext
+from ..core.models import TreeNode
+from ..net.http import DEFAULT_HEADERS, TIMEOUT
+
+#: total sitemap files fetched across the whole tree (loop / fan-out guard)
+MAX_SITEMAPS = 60
+#: total page URLs collected before we stop (bounds the tree size)
+MAX_URLS = 10000
+#: how deep a chain of nested <sitemapindex> files we follow
+MAX_DEPTH = 6
+
+
+class SitemapModule(ScanModule):
+    name = "sitemap"
+    label = "Sitemap"
+
+    async def run(self, ctx: ScanContext) -> TreeNode | None:
+        return await asyncio.to_thread(self._build, ctx)
+
+    # ---- build (blocking; runs in a thread) -------------------------------
+
+    def _build(self, ctx: ScanContext) -> TreeNode | None:
+        urls = self._collect(ctx.domain)
+        if not urls:
+            return None  # -> EMPTY
+        truncated = len(urls) >= MAX_URLS
+        return _url_tree(urls, truncated)
+
+    def _collect(self, domain: str) -> list[str]:
+        """Fetch every sitemap (recursing indexes) and return the page URLs found."""
+        visited: set[str] = set()
+        urls: list[str] = []
+        budget = [MAX_SITEMAPS]  # mutable counter shared across the recursion
+        for sitemap in self._discover(domain):
+            self._crawl(sitemap, visited, urls, budget, depth=0)
+            if len(urls) >= MAX_URLS:
+                break
+        seen: set[str] = set()  # dedupe, preserve order
+        return [u for u in urls if not (u in seen or seen.add(u))]
+
+    def _crawl(self, url: str, visited: set[str], urls: list[str], budget: list[int], depth: int) -> None:
+        if url in visited or budget[0] <= 0 or depth > MAX_DEPTH or len(urls) >= MAX_URLS:
+            return
+        visited.add(url)
+        budget[0] -= 1
+
+        raw = self._get(url)
+        if raw is None:
+            return
+        kind, locs = _parse_sitemap(raw)
+        if kind == "index":
+            for loc in locs:
+                self._crawl(loc, visited, urls, budget, depth + 1)
+                if len(urls) >= MAX_URLS:
+                    return
+        elif kind == "urlset":
+            for loc in locs:
+                urls.append(loc)
+                if len(urls) >= MAX_URLS:
+                    return
+
+    # ---- discovery + fetch ------------------------------------------------
+
+    def _discover(self, domain: str) -> list[str]:
+        """Sitemap URLs from robots.txt, else the two conventional defaults."""
+        urls = self._robots_sitemaps(domain)
+        if not urls:
+            base = f"https://{domain}"
+            urls = [f"{base}/sitemap.xml", f"{base}/sitemap_index.xml"]
+        seen: set[str] = set()
+        return [u for u in urls if not (u in seen or seen.add(u))]
+
+    @staticmethod
+    def _robots_sitemaps(domain: str) -> list[str]:
+        try:
+            resp = requests.get(
+                f"https://{domain}/robots.txt",
+                headers=DEFAULT_HEADERS, timeout=TIMEOUT, allow_redirects=True,
+            )
+            if resp.status_code != 200:
+                return []
+            return [
+                ln.split(":", 1)[1].strip()
+                for ln in resp.text.splitlines()
+                if ln.strip().lower().startswith("sitemap:")
+            ]
+        except Exception:  # noqa: BLE001
+            return []
+
+    @staticmethod
+    def _get(url: str) -> bytes | None:
+        """Fetch a sitemap; gunzip ``.xml.gz`` / gzip payloads by magic bytes."""
+        try:
+            resp = requests.get(url, headers=DEFAULT_HEADERS, timeout=TIMEOUT, allow_redirects=True)
+            if resp.status_code != 200:
+                return None
+            raw = resp.content
+            if raw[:2] == b"\x1f\x8b":  # gzip magic — a served .gz file (not transport enc.)
+                raw = gzip.decompress(raw)
+            return raw
+        except Exception:  # noqa: BLE001
+            return None
+
+
+# ---- parsing + tree building ----------------------------------------------
+
+
+def _parse_sitemap(raw: bytes) -> tuple[str | None, list[str]]:
+    """Return ``("index"|"urlset", [loc, …])`` or ``(None, [])`` if unparseable."""
+    try:
+        root = ET.fromstring(raw)
+    except ET.ParseError:
+        return None, []
+    locs = [
+        el.text.strip()
+        for el in root.iter()
+        if _localname(el.tag) == "loc" and el.text and el.text.strip()
+    ]
+    kind = "index" if _localname(root.tag) == "sitemapindex" else "urlset"
+    return kind, locs
+
+
+def _localname(tag: str) -> str:
+    """Strip the ``{namespace}`` prefix ElementTree keeps on namespaced tags."""
+    return tag.rsplit("}", 1)[-1].lower()
+
+
+def _segments(url: str, include_host: bool) -> list[str]:
+    """Path segments for `url` as tree labels: ``/blog``, ``/post`` … An empty path
+    (the homepage) yields ``[]`` — it *is* the ``/`` root, not a child of it. When
+    `include_host`, a leading host label (no slash) groups multi-host sitemaps. Any
+    query is kept on the final leaf."""
+    p = urlparse(url)
+    parts = [s for s in p.path.split("/") if s]
+    segs: list[str] = []
+    if include_host and p.netloc:
+        segs.append(p.netloc)
+    segs += ["/" + s for s in parts]
+    if p.query and segs:
+        segs[-1] = segs[-1] + "?" + p.query
+    return segs
+
+
+def _url_tree(urls: list[str], truncated: bool) -> TreeNode:
+    """Build a path-keyed tree under a visible ``/`` root. Within every level,
+    branches (toggleable folders) sort first, then leaf pages — both alphabetical."""
+    include_host = len({urlparse(u).netloc for u in urls if urlparse(u).netloc}) > 1
+    root = TreeNode(label="/")  # the site root / homepage; shown + expanded by the UI
+    index: dict[tuple[str, ...], TreeNode] = {(): root}
+    for url in sorted(urls):
+        parent = root
+        node = root
+        key: tuple[str, ...] = ()
+        for seg in _segments(url, include_host):
+            key += (seg,)
+            node = index.get(key)
+            if node is None:
+                node = TreeNode(label=seg)
+                index[key] = node
+                parent.children.append(node)
+            parent = node
+        node.url = url  # deepest node for this URL (a leaf, unless it later gains children)
+    if truncated:
+        root.children.append(TreeNode(label=f"… (truncated at {MAX_URLS} URLs)"))
+    root.total = len(urls)
+    _sort_tree(root)
+    return root
+
+
+def _sort_tree(node: TreeNode) -> None:
+    """Recursively order children: branches (have children) before leaves, each
+    group alphabetical by label."""
+    for child in node.children:
+        _sort_tree(child)
+    node.children.sort(key=lambda c: (not c.children, c.label.lower()))

{web_scanner-2.1.1 → web_scanner-2.1.3}/webscanner/modules/subdomains.py

@@ -4,7 +4,8 @@ Two native sources:
   1. Subject Alternative Names on the live TLS certificate (grabbed once in
      prefetch via ``ssl``/``socket``) — real hostnames the cert is valid for.
   2. Resolving a curated list of common subdomains via ``socket`` and keeping the
-     ones that exist.
+     ones that exist. Skipped when the domain has wildcard DNS (a random label
+     resolves), since every probe would otherwise be a false positive.
 
 Less exhaustive than a Certificate Transparency search, but instant and
 dependency-free. Swap in a CT/API source later if broader coverage is needed.
@@ -13,6 +14,7 @@ dependency-free. Swap in a CT/API source later if broader coverage is needed.
 from __future__ import annotations
 
 import asyncio
+import secrets
 import socket
 
 from ..core.module import ScanModule
@@ -22,6 +24,20 @@ COMMON_SUBDOMAINS = (
     "www", "mail", "webmail", "smtp", "imap", "pop", "ftp", "cpanel", "webdisk",
     "ns1", "ns2", "mx", "api", "dev", "staging", "blog", "shop", "admin",
     "portal", "vpn", "cdn", "m",
+    # SaaS & Authentication
+    "app", "login", "secure", "auth", "account", "dashboard",
+    # Support & Resources
+    "help", "support", "status", "docs", "download",
+    # Infrastructure & Internal
+    "test", "demo", "qa", "git", "internal", "autodiscover",
+    # Mail & DNS
+    "ns3", "mx1", "mx2", "email", "autoconfig", "owa", "exchange",
+    # Assets & CDN
+    "static", "assets", "media", "web",
+    # Environments & Dev infra
+    "beta", "sandbox", "gitlab", "wiki",
+    # Community & Customer
+    "store", "forum", "my", "mobile", "remote",
 )
 
 
@@ -41,7 +57,7 @@ class SubdomainsModule(ScanModule):
                 if host == ctx.domain or host.endswith("." + ctx.domain):
                     found.add(host)
 
-        # 2. resolve common subdomains concurrently
+        # 2. resolve common subdomains concurrently (skip if wildcard DNS)
         async def probe(sub: str) -> str | None:
             host = f"{sub}.{ctx.domain}"
             try:
@@ -50,7 +66,12 @@ class SubdomainsModule(ScanModule):
             except OSError:
                 return None
 
-        resolved = await asyncio.gather(*(probe(s) for s in COMMON_SUBDOMAINS))
-        found.update(host for host in resolved if host)
+        # A random label should not exist; if it resolves the domain has
+        # wildcard DNS and the brute-force list would be all false positives.
+        wildcard = await probe(f"wildcard-probe-{secrets.token_hex(6)}")
+        if wildcard is None:
+            resolved = await asyncio.gather(*(probe(s) for s in COMMON_SUBDOMAINS))
+            found.update(host for host in resolved if host)
+
         found.discard(ctx.domain)
         return sorted(found)

web_scanner-2.1.3/webscanner/modules/tech.py

@@ -0,0 +1,52 @@
+"""Technology detection via Wappalyzer.
+
+Uses ``analyze(..., scan_type="fast")`` — a pure-HTTP path (no headless browser) —
+which returns per-technology version, confidence, categories and groups. Rendered as
+one :class:`Grid` per *group* (Wappalyzer's high-level bucket, e.g. "Web servers",
+"Analytics"), stacked as titled :class:`Sections`. A technology in several groups
+appears in each of their tables.
+"""
+
+from __future__ import annotations
+
+import asyncio
+
+from wappalyzer import analyze
+
+from ..core.module import ScanModule
+from ..core.context import ScanContext
+from ..core.models import Grid, Section, Sections
+
+COLUMNS = ["Name", "Category", "Confidence", "Version"]
+#: fixed column widths so every per-group table on the tab lines up identically
+WIDTHS = [26, 22, 12, 12]
+#: bucket for technologies Wappalyzer reports without any group
+UNGROUPED = "Other"
+
+
+class TechModule(ScanModule):
+    name = "tech"
+    label = "Tech"
+
+    async def run(self, ctx: ScanContext) -> Sections:
+        def detect() -> Sections:
+            results = analyze(url=ctx.url, scan_type="fast", timeout=30)
+            techs = results.get(ctx.url) or next(iter(results.values()), {})
+            # group name -> rows; a tech lands in every group it declares
+            groups: dict[str, list[list[str]]] = {}
+            for name, info in techs.items():
+                row = [
+                    name,
+                    ", ".join(info.get("categories") or []) or "-",
+                    f"{info.get('confidence', 0)}%",
+                    info.get("version") or "-",
+                ]
+                for group in info.get("groups") or [UNGROUPED]:
+                    groups.setdefault(group, []).append(row)
+            sections = Sections()
+            for group in sorted(groups, key=str.lower):  # tables A→Z by group
+                rows = sorted(groups[group], key=lambda r: r[0].lower())  # names A→Z
+                sections.append(Section(group, Grid(COLUMNS, rows, widths=WIDTHS)))
+            return sections
+
+        return await asyncio.to_thread(detect)