web-scanner 2.1.3__tar.gz → 2.1.4__tar.gz

{web_scanner-2.1.3 → web_scanner-2.1.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: web-scanner
-Version: 2.1.3
+Version: 2.1.4
 Summary: An async Textual TUI for website reconnaissance — DNS, WHOIS, TLS, security, tech-stack and SEO, with no paid APIs.
 Project-URL: Homepage, https://github.com/iamramizk/web-scanner
 Project-URL: Repository, https://github.com/iamramizk/web-scanner
@@ -44,29 +44,49 @@ Requires-Dist: requests>=2.28.2
 Requires-Dist: rich>=15.0.0
 Requires-Dist: textual>=8.2.8
 Requires-Dist: wappalyzer>=2.0.2
+Provides-Extra: test
+Requires-Dist: pytest>=8.0; extra == 'test'
 Description-Content-Type: text/markdown
 
+<div align="center">
+
 # WebScanner
 
-An async [Textual](https://textual.textualize.io/) TUI for website reconnaissance. Point it
-at a domain and it concurrently gathers DNS, WHOIS, TLS, security, tech-stack and SEO
-intelligence, then lays it out across tabs — with a live country map and a server-status
-panel that stay pinned in place.
+_An async Textual TUI for website reconnaissance — DNS, WHOIS, TLS, security, tech-stack and SEO, with no paid APIs._
+
+[![Python](https://img.shields.io/badge/Python-3.11%2B-3776AB?style=for-the-badge&logo=python&logoColor=white)](https://www.python.org/downloads/)
+[![PyPI Downloads](https://img.shields.io/pypi/dm/web-scanner?style=for-the-badge&logo=pypi&logoColor=white&color=006DAD)](https://pypi.org/project/web-scanner/)
+[![Release](https://img.shields.io/github/v/release/iamramizk/web-scanner?style=for-the-badge&logo=github&color=2EA44F)](https://github.com/iamramizk/web-scanner/releases/latest)
+
+</div>
+
+Point it at a domain and it concurrently gathers DNS, WHOIS, TLS, security, tech-stack and
+SEO intelligence, then lays it out across tabs — with a live activity log, a country map and
+a server-status panel that stay pinned in place.
 
 **No paid APIs.** Everything runs off free, public endpoints and the standard library:
 `ip-api.com` for geolocation, Cloudflare/Google DoH and `dig` against public filtering
 resolvers for blocklist checks, the system `whois`, and stdlib `ssl`/`socket`. Country
 borders are embedded (Natural Earth), so even the map needs no tile service.
 
-![WebScanner](https://raw.githubusercontent.com/iamramizk/web-scanner/main/.github/screenshot-v2.png)
+![WebScanner](https://raw.githubusercontent.com/iamramizk/web-scanner/main/.github/screenshot-v2.1.png)
+
+## Contents
+
+- [Features](#features)
+- [Installation](#installation)
+  - [From source (development)](#from-source-development)
+- [Usage](#usage)
+- [Updating](#updating)
+- [Keys](#keys)
 
 ## Features
 
 Ten tabs, scanned concurrently and rendered live as each module finishes:
 
 - **DNS** — A / AAAA / NS / CNAME / SOA / MX / TXT / CAA / DS / DNSKEY records, plus email
-  authentication folded in: DMARC (`_dmarc`) and DKIM (probes ~40 common selectors, each
-  labelled with its email provider).
+  authentication folded in: DMARC (`_dmarc`) and DKIM (probes ~40 known selectors, each hit
+  labelled with the email provider it points to, where the selector identifies one).
 - **Whois** — parsed system `whois`, with rich gTLD and ccTLD field support (registrar,
   dates, nameservers, per-contact details).
 - **Subdomains** — discovered natively from TLS certificate SANs and `socket` probes of
@@ -78,19 +98,28 @@ Ten tabs, scanned concurrently and rendered live as each module finishes:
   (AdGuard, CleanBrowsing, Cloudflare, Google, OpenDNS, Quad9).
 - **Headers** — the full set of HTTP response headers.
 - **Tech** — technology-stack detection via [Wappalyzer](https://github.com/tunetheweb/wappalyzer),
-  split into one table per technology group, each showing name, category, confidence and version.
+  listing each technology with its category, confidence, groups and version.
 - **SEO** — page content (title/description with length hints, H1–H3, social links), top
   keyword n-grams, `robots.txt` and sitemaps, and JSON-LD structured data.
 - **Sitemap** — the site's URLs discovered from its `sitemap.xml` (recursing into nested
   sitemap indexes) and rebuilt into a clickable URL-path tree — no crawling.
 - **Links** — internal and external links, with their anchor text.
 
-Alongside the tabs, two fixed panels:
+Alongside the tabs, three fixed panels:
 
+- **Activity Log** — a running narration of the scan: what each module found, what came back
+  empty and what broke, so every headline result is visible without switching tabs.
 - **Country map** — real country outlines auto-framed around the server's location, drawn
   with braille characters (`+` / `-` to zoom).
-- **Server** — online status and response time, IP, geolocation, ISP, AS, hosting provider
-  and detected CMS (name and version, from the Tech scan).
+- **Server** — online status and response time, the final URL after redirects, IP,
+  geolocation, ISP, AS, hosting provider and detected CMS (name and version).
+
+Requests to the site being scanned wear a **coherent desktop-Chrome identity** — a real
+User-Agent plus the headers Chrome actually sends beside it — picked once per scan and
+reused for every request, and reported in the Activity Log. This is about getting the real
+page back rather than a WAF block page, and it only defeats naive User-Agent filtering: the
+TLS and HTTP/2 fingerprints are still those of `requests`, so enterprise bot management sees
+straight through it. Third-party lookups (ip-api, DoH) keep an honest scanner User-Agent.
 
 ## Installation
 
@@ -143,7 +172,7 @@ so reinstall from source instead:
 pipx install --force git+https://github.com/iamramizk/web-scanner
 ```
 
-### Keys
+## Keys
 
 | Key              | Action                                                              |
 | ---------------- | ------------------------------------------------------------------- |
@@ -151,6 +180,6 @@ pipx install --force git+https://github.com/iamramizk/web-scanner
 | `+` / `-`        | Zoom the country map in / out                                       |
 | `↑` / `↓` `enter`| Navigate the Sitemap tree (`space` expands/collapses all)          |
 | `r`              | Rescan                                                              |
-| `s`              | Save — export every tab to CSV under `output/<domain>_<timestamp>/` |
+| `s`              | Save — export every tab to CSV under `<domain>_<timestamp>/`        |
 | `esc`            | Edit the domain and scan a new one                                  |
 | `q`              | Quit                                                                |

{web_scanner-2.1.3 → web_scanner-2.1.4}/README.md

@@ -1,24 +1,42 @@
+<div align="center">
+
 # WebScanner
 
-An async [Textual](https://textual.textualize.io/) TUI for website reconnaissance. Point it
-at a domain and it concurrently gathers DNS, WHOIS, TLS, security, tech-stack and SEO
-intelligence, then lays it out across tabs — with a live country map and a server-status
-panel that stay pinned in place.
+_An async Textual TUI for website reconnaissance — DNS, WHOIS, TLS, security, tech-stack and SEO, with no paid APIs._
+
+[![Python](https://img.shields.io/badge/Python-3.11%2B-3776AB?style=for-the-badge&logo=python&logoColor=white)](https://www.python.org/downloads/)
+[![PyPI Downloads](https://img.shields.io/pypi/dm/web-scanner?style=for-the-badge&logo=pypi&logoColor=white&color=006DAD)](https://pypi.org/project/web-scanner/)
+[![Release](https://img.shields.io/github/v/release/iamramizk/web-scanner?style=for-the-badge&logo=github&color=2EA44F)](https://github.com/iamramizk/web-scanner/releases/latest)
+
+</div>
+
+Point it at a domain and it concurrently gathers DNS, WHOIS, TLS, security, tech-stack and
+SEO intelligence, then lays it out across tabs — with a live activity log, a country map and
+a server-status panel that stay pinned in place.
 
 **No paid APIs.** Everything runs off free, public endpoints and the standard library:
 `ip-api.com` for geolocation, Cloudflare/Google DoH and `dig` against public filtering
 resolvers for blocklist checks, the system `whois`, and stdlib `ssl`/`socket`. Country
 borders are embedded (Natural Earth), so even the map needs no tile service.
 
-![WebScanner](https://raw.githubusercontent.com/iamramizk/web-scanner/main/.github/screenshot-v2.png)
+![WebScanner](https://raw.githubusercontent.com/iamramizk/web-scanner/main/.github/screenshot-v2.1.png)
+
+## Contents
+
+- [Features](#features)
+- [Installation](#installation)
+  - [From source (development)](#from-source-development)
+- [Usage](#usage)
+- [Updating](#updating)
+- [Keys](#keys)
 
 ## Features
 
 Ten tabs, scanned concurrently and rendered live as each module finishes:
 
 - **DNS** — A / AAAA / NS / CNAME / SOA / MX / TXT / CAA / DS / DNSKEY records, plus email
-  authentication folded in: DMARC (`_dmarc`) and DKIM (probes ~40 common selectors, each
-  labelled with its email provider).
+  authentication folded in: DMARC (`_dmarc`) and DKIM (probes ~40 known selectors, each hit
+  labelled with the email provider it points to, where the selector identifies one).
 - **Whois** — parsed system `whois`, with rich gTLD and ccTLD field support (registrar,
   dates, nameservers, per-contact details).
 - **Subdomains** — discovered natively from TLS certificate SANs and `socket` probes of
@@ -30,19 +48,28 @@ Ten tabs, scanned concurrently and rendered live as each module finishes:
   (AdGuard, CleanBrowsing, Cloudflare, Google, OpenDNS, Quad9).
 - **Headers** — the full set of HTTP response headers.
 - **Tech** — technology-stack detection via [Wappalyzer](https://github.com/tunetheweb/wappalyzer),
-  split into one table per technology group, each showing name, category, confidence and version.
+  listing each technology with its category, confidence, groups and version.
 - **SEO** — page content (title/description with length hints, H1–H3, social links), top
   keyword n-grams, `robots.txt` and sitemaps, and JSON-LD structured data.
 - **Sitemap** — the site's URLs discovered from its `sitemap.xml` (recursing into nested
   sitemap indexes) and rebuilt into a clickable URL-path tree — no crawling.
 - **Links** — internal and external links, with their anchor text.
 
-Alongside the tabs, two fixed panels:
+Alongside the tabs, three fixed panels:
 
+- **Activity Log** — a running narration of the scan: what each module found, what came back
+  empty and what broke, so every headline result is visible without switching tabs.
 - **Country map** — real country outlines auto-framed around the server's location, drawn
   with braille characters (`+` / `-` to zoom).
-- **Server** — online status and response time, IP, geolocation, ISP, AS, hosting provider
-  and detected CMS (name and version, from the Tech scan).
+- **Server** — online status and response time, the final URL after redirects, IP,
+  geolocation, ISP, AS, hosting provider and detected CMS (name and version).
+
+Requests to the site being scanned wear a **coherent desktop-Chrome identity** — a real
+User-Agent plus the headers Chrome actually sends beside it — picked once per scan and
+reused for every request, and reported in the Activity Log. This is about getting the real
+page back rather than a WAF block page, and it only defeats naive User-Agent filtering: the
+TLS and HTTP/2 fingerprints are still those of `requests`, so enterprise bot management sees
+straight through it. Third-party lookups (ip-api, DoH) keep an honest scanner User-Agent.
 
 ## Installation
 
@@ -95,7 +122,7 @@ so reinstall from source instead:
 pipx install --force git+https://github.com/iamramizk/web-scanner
 ```
 
-### Keys
+## Keys
 
 | Key              | Action                                                              |
 | ---------------- | ------------------------------------------------------------------- |
@@ -103,6 +130,6 @@ pipx install --force git+https://github.com/iamramizk/web-scanner
 | `+` / `-`        | Zoom the country map in / out                                       |
 | `↑` / `↓` `enter`| Navigate the Sitemap tree (`space` expands/collapses all)          |
 | `r`              | Rescan                                                              |
-| `s`              | Save — export every tab to CSV under `output/<domain>_<timestamp>/` |
+| `s`              | Save — export every tab to CSV under `<domain>_<timestamp>/`        |
 | `esc`            | Edit the domain and scan a new one                                  |
 | `q`              | Quit                                                                |

{web_scanner-2.1.3 → web_scanner-2.1.4}/pyproject.toml

@@ -32,6 +32,11 @@ dependencies = [
     "beautifulsoup4>=4.12.2",
 ]
 
+[project.optional-dependencies]
+# Test-only; not a runtime dep, so requirements.txt does not mirror it.
+# pip install -e ".[test]" && python -m pytest
+test = ["pytest>=8.0"]
+
 [project.urls]
 Homepage = "https://github.com/iamramizk/web-scanner"
 Repository = "https://github.com/iamramizk/web-scanner"

{web_scanner-2.1.3 → web_scanner-2.1.4}/webscanner/__init__.py

@@ -1,3 +1,3 @@
 """web-scanner v2 — async website reconnaissance TUI package."""
 
-__version__ = "2.1.3"
+__version__ = "2.1.4"

web_scanner-2.1.4/webscanner/core/context.py

@@ -0,0 +1,72 @@
+"""Shared per-scan context passed to every module.
+
+The orchestrator's prefetch phase populates the shared network fields (one DNS
+resolve, one HTTP GET, one TLS handshake, one geo lookup) so individual modules
+reuse them instead of refetching.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from typing import Any
+from urllib.parse import urlparse
+
+from .. import helpers
+from ..net.agents import Profile, random_profile
+
+
+@dataclass
+class ScanContext:
+    # inputs
+    domain: str  # bare host, e.g. "example.com"
+    url: str  # full url, e.g. "https://example.com"
+
+    #: The browser identity every target-facing request wears. Chosen once per scan
+    #: and shared, so the site sees one consistent visitor rather than a different
+    #: browser per request — see net/agents.py. Not used for ip-api/DoH.
+    profile: Profile = field(default_factory=random_profile)
+
+    # shared prefetch results (filled by AsyncScanner.prefetch)
+    ip: str | None = None
+    html: str | None = None
+    headers: dict[str, str] = field(default_factory=dict)
+    status_code: int | None = None
+    response_time_ms: float | None = None
+    final_url: str | None = None
+    redirect_status: str | None = None  # e.g. "301 Moved Permanently" for cross-domain redirects
+    tls_cert: dict[str, Any] | None = None
+    geo: dict[str, Any] | None = None
+    fetch_error: str | None = None
+
+    @classmethod
+    def from_target(cls, target: str) -> "ScanContext":
+        domain, url = helpers.normalise(target)
+        return cls(domain=domain, url=url)
+
+    @property
+    def base(self) -> str:
+        """Origin to build target URLs from: ``<scheme>://<domain>``.
+
+        The scheme is the one prefetch actually *reached*, not the one we asked for.
+        ``helpers.normalise()`` always hands us ``https://`` regardless of what the
+        user typed, so :attr:`url` is not a scheme signal; ``http.fetch()`` retries
+        over ``http://`` when https won't connect, and :attr:`final_url` records
+        where it landed. An http-only site that builds its URLs from :attr:`url`
+        silently gets nothing back (no robots.txt, no sitemap, no tech) while the
+        rest of the scan succeeds.
+
+        The host comes from :attr:`domain`, *not* from ``final_url``: a bare→www
+        redirect would otherwise pin us to ``www.example.com``, and requests we
+        build follow redirects there anyway.
+
+        **Only valid after prefetch.** Before it (``final_url`` is None) this falls
+        back to https — which is also the right answer when the fetch failed
+        outright, since nothing else will connect either. ``prefetch()`` itself must
+        never read this: it is what determines the scheme.
+        """
+        scheme = urlparse(self.final_url).scheme if self.final_url else ""
+        return f"{scheme or 'https'}://{self.domain}"
+
+    @property
+    def online(self) -> bool:
+        return self.status_code is not None and self.fetch_error is None

{web_scanner-2.1.3 → web_scanner-2.1.4}/webscanner/core/scanner.py

@@ -56,7 +56,7 @@ class AsyncScanner:
 
         async def _fetch() -> None:
             try:
-                r = await asyncio.to_thread(http.fetch, self.ctx.url)
+                r = await asyncio.to_thread(http.fetch, self.ctx.url, self.ctx.profile)
                 self.ctx.status_code = r["status_code"]
                 self.ctx.headers = r["headers"]
                 self.ctx.html = r["html"]

{web_scanner-2.1.3 → web_scanner-2.1.4}/webscanner/modules/seo.py

@@ -26,7 +26,8 @@ from ..colors import GREEN, RED, MUTED
 from ..core.module import ScanModule
 from ..core.context import ScanContext
 from ..core.models import Section, Sections
-from ..net.http import DEFAULT_HEADERS, TIMEOUT
+from ..net.agents import Profile
+from ..net.http import TIMEOUT
 
 TITLE_RANGE = (30, 60)
 DESC_RANGE = (70, 160)
@@ -103,7 +104,7 @@ class SeoModule(ScanModule):
     label = "SEO"
 
     async def run(self, ctx: ScanContext) -> Sections:
-        robots_coro = asyncio.to_thread(self._fetch_robots, ctx.domain)
+        robots_coro = asyncio.to_thread(self._fetch_robots, ctx.base, ctx.profile)
         if not ctx.html:
             robots = await robots_coro
             note = {"note": "no page content"}
@@ -169,11 +170,12 @@ class SeoModule(ScanModule):
         return schema, content, keywords
 
     @staticmethod
-    def _fetch_robots(domain: str) -> dict[str, object]:
+    def _fetch_robots(base: str, profile: Profile) -> dict[str, object]:
+        url = f"{base}/robots.txt"
         try:
             resp = requests.get(
-                f"https://{domain}/robots.txt",
-                headers=DEFAULT_HEADERS, timeout=TIMEOUT, allow_redirects=True,
+                url,
+                headers=profile.headers(url), timeout=TIMEOUT, allow_redirects=True,
             )
             text = resp.text.strip()
             ctype = resp.headers.get("content-type", "").lower()

{web_scanner-2.1.3 → web_scanner-2.1.4}/webscanner/modules/sitemap.py

@@ -27,7 +27,8 @@ import requests
 from ..core.module import ScanModule
 from ..core.context import ScanContext
 from ..core.models import TreeNode
-from ..net.http import DEFAULT_HEADERS, TIMEOUT
+from ..net.agents import Profile
+from ..net.http import TIMEOUT
 
 #: total sitemap files fetched across the whole tree (loop / fan-out guard)
 MAX_SITEMAPS = 60
@@ -47,37 +48,40 @@ class SitemapModule(ScanModule):
     # ---- build (blocking; runs in a thread) -------------------------------
 
     def _build(self, ctx: ScanContext) -> TreeNode | None:
-        urls = self._collect(ctx.domain)
+        urls = self._collect(ctx.base, ctx.profile)
         if not urls:
             return None  # -> EMPTY
         truncated = len(urls) >= MAX_URLS
         return _url_tree(urls, truncated)
 
-    def _collect(self, domain: str) -> list[str]:
+    def _collect(self, base: str, profile: Profile) -> list[str]:
         """Fetch every sitemap (recursing indexes) and return the page URLs found."""
         visited: set[str] = set()
         urls: list[str] = []
         budget = [MAX_SITEMAPS]  # mutable counter shared across the recursion
-        for sitemap in self._discover(domain):
-            self._crawl(sitemap, visited, urls, budget, depth=0)
+        for sitemap in self._discover(base, profile):
+            self._crawl(sitemap, profile, visited, urls, budget, depth=0)
             if len(urls) >= MAX_URLS:
                 break
         seen: set[str] = set()  # dedupe, preserve order
         return [u for u in urls if not (u in seen or seen.add(u))]
 
-    def _crawl(self, url: str, visited: set[str], urls: list[str], budget: list[int], depth: int) -> None:
+    def _crawl(
+        self, url: str, profile: Profile, visited: set[str], urls: list[str],
+        budget: list[int], depth: int,
+    ) -> None:
         if url in visited or budget[0] <= 0 or depth > MAX_DEPTH or len(urls) >= MAX_URLS:
             return
         visited.add(url)
         budget[0] -= 1
 
-        raw = self._get(url)
+        raw = self._get(url, profile)
         if raw is None:
             return
         kind, locs = _parse_sitemap(raw)
         if kind == "index":
             for loc in locs:
-                self._crawl(loc, visited, urls, budget, depth + 1)
+                self._crawl(loc, profile, visited, urls, budget, depth + 1)
                 if len(urls) >= MAX_URLS:
                     return
         elif kind == "urlset":
@@ -88,21 +92,21 @@ class SitemapModule(ScanModule):
 
     # ---- discovery + fetch ------------------------------------------------
 
-    def _discover(self, domain: str) -> list[str]:
+    def _discover(self, base: str, profile: Profile) -> list[str]:
         """Sitemap URLs from robots.txt, else the two conventional defaults."""
-        urls = self._robots_sitemaps(domain)
+        urls = self._robots_sitemaps(base, profile)
         if not urls:
-            base = f"https://{domain}"
             urls = [f"{base}/sitemap.xml", f"{base}/sitemap_index.xml"]
         seen: set[str] = set()
         return [u for u in urls if not (u in seen or seen.add(u))]
 
     @staticmethod
-    def _robots_sitemaps(domain: str) -> list[str]:
+    def _robots_sitemaps(base: str, profile: Profile) -> list[str]:
+        url = f"{base}/robots.txt"
         try:
             resp = requests.get(
-                f"https://{domain}/robots.txt",
-                headers=DEFAULT_HEADERS, timeout=TIMEOUT, allow_redirects=True,
+                url,
+                headers=profile.headers(url), timeout=TIMEOUT, allow_redirects=True,
             )
             if resp.status_code != 200:
                 return []
@@ -115,10 +119,12 @@ class SitemapModule(ScanModule):
             return []
 
     @staticmethod
-    def _get(url: str) -> bytes | None:
+    def _get(url: str, profile: Profile) -> bytes | None:
         """Fetch a sitemap; gunzip ``.xml.gz`` / gzip payloads by magic bytes."""
         try:
-            resp = requests.get(url, headers=DEFAULT_HEADERS, timeout=TIMEOUT, allow_redirects=True)
+            resp = requests.get(
+                url, headers=profile.headers(url), timeout=TIMEOUT, allow_redirects=True
+            )
             if resp.status_code != 200:
                 return None
             raw = resp.content

web_scanner-2.1.4/webscanner/modules/tech.py

@@ -0,0 +1,119 @@
+"""Technology detection via Wappalyzer.
+
+Uses ``analyze(..., scan_type="fast")`` — a pure-HTTP path (no headless browser) —
+which returns per-technology version, confidence, categories and groups. Rendered as
+one :class:`Grid` per *group* (Wappalyzer's high-level bucket, e.g. "Web servers",
+"Analytics"), stacked as titled :class:`Sections`. A technology in several groups
+appears in each of their tables.
+
+Wappalyzer fetches the page *itself* — this is the one target-facing request the scan
+doesn't make through ``net.http`` — and ``analyze()`` takes no header argument, so its
+UA is patched in below.
+"""
+
+from __future__ import annotations
+
+import asyncio
+
+import requests
+from wappalyzer import analyze
+
+from ..core.module import ScanModule
+from ..core.context import ScanContext
+from ..core.models import Grid, Section, Sections
+from ..net.agents import Profile
+
+COLUMNS = ["Name", "Category", "Confidence", "Version"]
+#: fixed column widths so every per-group table on the tab lines up identically
+WIDTHS = [26, 22, 12, 12]
+#: bucket for technologies Wappalyzer reports without any group
+UNGROUPED = "Other"
+
+
+class _UARewrite:
+    """Stands in for the ``requests`` module inside ``wappalyzer.core.requester``.
+
+    Wappalyzer hardcodes its own User-Agent (a Chrome on **X11; Linux**) in a local
+    dict inside ``get_response``, and exposes no way to override it. Left alone, the
+    target sees two visitors from one IP in the same second — our profile, plus a
+    Linux desktop Chrome — which is precisely the inconsistency the profile exists to
+    avoid, and Linux desktop is the rarer claim of the two.
+
+    Why swap the module's ``requests`` attribute rather than patch ``get_response``:
+    ``core/analyzer.py`` does ``from ...requester import get_response``, binding the
+    function into its own namespace at import time, so patching ``requester.get_response``
+    would silently do nothing. The function *body*, though, looks up its module-global
+    ``requests`` on every call — so replacing that attribute catches the call wherever
+    the function was imported from.
+
+    Only the User-Agent key is rewritten; the rest of Wappalyzer's header block is
+    passed through untouched. That is deliberate — its ``Accept-Encoding: deflate`` is
+    load-bearing (see ``net/agents.py`` on advertising encodings you can't decode), and
+    passing its dict through rather than restating it means their header block can
+    change without us diverging from it.
+
+    Scoped to Wappalyzer's own module attribute, so our ``requests`` calls are
+    unaffected. Re-applying per scan never nests wrappers: ``_real`` is the genuine
+    module, captured here, not whatever currently sits on ``requester.requests``.
+    """
+
+    def __init__(self, user_agent: str) -> None:
+        self._user_agent = user_agent
+        self._real = requests
+
+    def get(self, url: str, **kwargs: object):
+        headers = {**(kwargs.pop("headers", None) or {}), "User-Agent": self._user_agent}
+        return self._real.get(url, headers=headers, **kwargs)
+
+    def __getattr__(self, name: str) -> object:
+        # requester also reaches for requests.exceptions.RequestException.
+        return getattr(requests, name)
+
+
+def _wear_profile(profile: Profile) -> None:
+    """Point Wappalyzer's request at the scan's identity.
+
+    Guarded end-to-end: ``wappalyzer.core.requester`` is a private path that a version
+    bump could move or restructure. If it isn't where we expect, the tech module simply
+    scans with Wappalyzer's own UA — today's behaviour — rather than failing the tab.
+    """
+    try:
+        from wappalyzer.core import requester
+
+        requester.requests = _UARewrite(profile.user_agent)
+    except Exception:  # noqa: BLE001 - a moved internal must not cost us the tab
+        pass
+
+
+class TechModule(ScanModule):
+    name = "tech"
+    label = "Tech"
+
+    async def run(self, ctx: ScanContext) -> Sections:
+        _wear_profile(ctx.profile)
+
+        def detect() -> Sections:
+            # ctx.base, not ctx.url: Wappalyzer fetches the page itself, so on an
+            # http-only site an https:// url hands it a connection error and the tab
+            # comes back empty. It also keeps what Wappalyzer sees in step with the
+            # ctx.html that seo/links/the CMS generator path already read.
+            results = analyze(url=ctx.base, scan_type="fast", timeout=30)
+            techs = results.get(ctx.base) or next(iter(results.values()), {})
+            # group name -> rows; a tech lands in every group it declares
+            groups: dict[str, list[list[str]]] = {}
+            for name, info in techs.items():
+                row = [
+                    name,
+                    ", ".join(info.get("categories") or []) or "-",
+                    f"{info.get('confidence', 0)}%",
+                    info.get("version") or "-",
+                ]
+                for group in info.get("groups") or [UNGROUPED]:
+                    groups.setdefault(group, []).append(row)
+            sections = Sections()
+            for group in sorted(groups, key=str.lower):  # tables A→Z by group
+                rows = sorted(groups[group], key=lambda r: r[0].lower())  # names A→Z
+                sections.append(Section(group, Grid(COLUMNS, rows, widths=WIDTHS)))
+            return sections
+
+        return await asyncio.to_thread(detect)

web_scanner-2.1.4/webscanner/net/agents.py

@@ -0,0 +1,153 @@
+"""Browser identities for target-facing requests.
+
+One :class:`Profile` is one coherent desktop-Chrome identity: a User-Agent plus the
+headers that Chrome genuinely sends beside it. They live together because a UA on its
+own is *worse* than no disguise — ``requests`` defaults to ``Accept: */*`` and no
+``Sec-Fetch-*``/``Sec-CH-UA``, and a "Chrome" that sends those is trivially spotted by
+the mid-tier WAFs that then serve a block page instead of the site. Getting real
+content back is the point; looking natural is how.
+
+Scope: **target-facing requests only** (homepage, robots.txt, sitemaps). ip-api and
+Cloudflare DoH keep the honest ``http.USER_AGENT`` — they don't block, don't care, and
+a browser UA beside ``accept: application/dns-json`` would be incoherent anyway.
+
+Why the pool is all recent desktop Chrome, and not a browser zoo: the scan picks one
+profile and reuses it, so rotation only ever varies things *between* scans. Sites do
+serve different markup to Safari vs Firefox, so a mixed pool would make two scans of
+one domain disagree — a reliability cost paid for a stealth benefit that isn't real,
+since the IP and TLS fingerprint are identical across scans regardless. Chrome N vs
+N-2 on Windows vs macOS is variation *below the threshold where sites branch*, so
+rotation costs nothing here. Chrome-on-Windows is also the most common UA on earth:
+the most boring thing you can claim. Pin ``CHROME_VERSIONS``/``PLATFORMS`` to one
+entry each if you ever want scans fully deterministic.
+
+Known ceiling: this controls headers, not the TLS (JA3/JA4) or HTTP/2 fingerprint.
+``requests`` speaks HTTP/1.1 with an OpenSSL handshake no Chrome would produce, so
+against enterprise bot management (Cloudflare, Akamai) the disguise does not hold —
+those read the handshake, not the string. This buys the naive-UA-filter case, which is
+nearly all of them. Header *order* is likewise not ours to set (requests merges its own
+session defaults in first). Don't chase either through this file; the lever there is a
+TLS-impersonating client such as curl_cffi, which is a compiled dependency and a much
+bigger trade.
+"""
+
+from __future__ import annotations
+
+import random
+from dataclasses import dataclass
+
+#: Chrome stable majors, newest first — current + two back, which is roughly how the
+#: real Chrome population spreads across a release window.
+#:
+#: **Checked against the Chrome release API on 2026-07-17 (stable was 151).** These
+#: rot: a UA claiming a Chrome that shipped 18 months ago is a louder signal than an
+#: honest scanner UA, which defeats the whole file. Re-check when you next touch this:
+#: https://versionhistory.googleapis.com/v1/chrome/platforms/win/channels/stable/versions
+CHROME_VERSIONS: tuple[str, ...] = ("151", "150", "149")
+
+#: (UA platform token, sec-ch-ua-platform value). Both are frozen strings in modern
+#: Chrome: its reduced UA reports every macOS as "10_15_7" and every Windows as
+#: "Windows NT 10.0" no matter the real OS, so these are correct as literals rather
+#: than lazily hardcoded.
+PLATFORMS: tuple[tuple[str, str], ...] = (
+    ("Windows NT 10.0; Win64; x64", "Windows"),
+    ("Macintosh; Intel Mac OS X 10_15_7", "macOS"),
+)
+
+#: What Chrome sends for a top-level navigation. A homepage/robots.txt/sitemap.xml GET
+#: is exactly what you'd get by typing the URL in the bar, so one navigation header set
+#: covers all three honestly.
+_ACCEPT = (
+    "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,"
+    "image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"
+)
+
+#: Real Chrome 151 sends "gzip, deflate, br, zstd". We deliberately advertise less.
+#:
+#: This is the one place we knowingly diverge from the browser, and it is not an
+#: oversight: brotli needs the `brotli` package and zstd needs `zstandard` (or a
+#: Python 3.14 stdlib that `requires-python = ">=3.11"` does not promise). Advertise an
+#: encoding we can't decode and the body comes back as bytes-shaped garbage — which
+#: silently poisons ctx.html and, with it, seo, links and tech. A hypothetical
+#: fingerprinter noticing the missing `br` beats a guaranteed corrupted scan.
+#:
+#: If you ever want the profile fully coherent, add `brotli` as a dependency and put
+#: `br` back — don't add `br` without it.
+_ACCEPT_ENCODING = "gzip, deflate"
+
+
+@dataclass(frozen=True)
+class Profile:
+    """One browser identity, chosen once per scan and reused for every request to the
+    target — a real browser keeps one identity per session. Rotating per *request*
+    would be the louder tell: one IP fetching `/` as Chrome and 60 sitemaps as Safari
+    within a few seconds is not a thing a browser does."""
+
+    chrome: str
+    """Chrome major version, e.g. "151"."""
+
+    ua_platform: str
+    """The platform token inside the UA string."""
+
+    platform: str
+    """The `sec-ch-ua-platform` value, e.g. "Windows"."""
+
+    @property
+    def user_agent(self) -> str:
+        # Chrome freezes the minor version at 0.0.0 in its reduced UA — the real build
+        # number (151.0.7922.34) never appears, so writing one would be the error.
+        return (
+            f"Mozilla/5.0 ({self.ua_platform}) AppleWebKit/537.36 (KHTML, like Gecko) "
+            f"Chrome/{self.chrome}.0.0.0 Safari/537.36"
+        )
+
+    @property
+    def label(self) -> str:
+        """Short human form for the activity log — "Chrome 151 · Windows"."""
+        return f"Chrome {self.chrome} · {self.platform}"
+
+    def headers(self, url: str) -> dict[str, str]:
+        """The full header set for a GET of ``url``.
+
+        Client hints are dropped for plaintext ``http://``: they're a secure-context
+        feature, so real Chrome doesn't send them there and doing so would contradict
+        the very UA they're meant to corroborate. This matters because ``http.fetch``
+        retries over http:// when TLS fails.
+        """
+        headers = {
+            "User-Agent": self.user_agent,
+            "Accept": _ACCEPT,
+            "Accept-Language": "en-US,en;q=0.9",
+            "Accept-Encoding": _ACCEPT_ENCODING,
+            "Upgrade-Insecure-Requests": "1",
+            "Sec-Fetch-Dest": "document",
+            "Sec-Fetch-Mode": "navigate",
+            "Sec-Fetch-Site": "none",
+            "Sec-Fetch-User": "?1",
+            "Connection": "keep-alive",
+        }
+        if url.lower().startswith("https://"):
+            headers |= {
+                # The "Not?A_Brand" entry is GREASE: deliberately arbitrary padding
+                # Chrome varies to stop anyone parsing this list rigidly. Any plausible
+                # value is as correct as another — it is designed to be ignored.
+                "sec-ch-ua": (
+                    f'"Chromium";v="{self.chrome}", '
+                    f'"Google Chrome";v="{self.chrome}", '
+                    f'"Not?A_Brand";v="24"'
+                ),
+                "sec-ch-ua-mobile": "?0",
+                "sec-ch-ua-platform": f'"{self.platform}"',
+            }
+        return headers
+
+
+def random_profile() -> Profile:
+    """Pick one identity for a scan. See the module docstring for why the pool is
+    narrow — the randomness is confined to axes sites don't branch on."""
+    ua_platform, platform = random.choice(PLATFORMS)
+    return Profile(
+        chrome=random.choice(CHROME_VERSIONS),
+        ua_platform=ua_platform,
+        platform=platform,
+    )

{web_scanner-2.1.3 → web_scanner-2.1.4}/webscanner/net/http.py

@@ -1,8 +1,21 @@
 """Blocking HTTP / DNS / TLS primitives.
 
 Kept synchronous on purpose: the async core calls these via ``asyncio.to_thread``.
-Centralising them here gives every module one user-agent, one timeout policy and
-one place to add retries/proxying later.
+Centralising them here gives every module one timeout policy and one place to add
+retries/proxying later.
+
+Two identities live here, and the split is deliberate:
+
+* **Target-facing** requests (homepage, robots.txt, sitemaps) wear the scan's
+  :class:`~webscanner.net.agents.Profile` — a browser identity, passed in by the
+  caller from ``ctx.profile``. See ``agents.py`` for why.
+* **Third-party APIs** (ip-api, Cloudflare DoH) use :data:`API_HEADERS`, which
+  identifies the tool honestly. They don't block and don't care, and there's no
+  reason to wear a costume for a free service we depend on.
+
+``API_HEADERS`` is named for that job rather than "default" on purpose: there is no
+sensible default here, and a name like ``DEFAULT_HEADERS`` is exactly what someone
+reaches for by accident on a target request.
 """
 
 from __future__ import annotations
@@ -15,14 +28,16 @@ from urllib.parse import urljoin, urlparse
 
 import requests
 
+from .agents import Profile
 from .psl import registrable_domain
 
 USER_AGENT = "web-scanner/2.0 (+https://github.com/iamramizk/web-scanner)"
 TIMEOUT = 12
-DEFAULT_HEADERS = {"User-Agent": USER_AGENT}
+#: For third-party APIs only — never for the target. See the module docstring.
+API_HEADERS = {"User-Agent": USER_AGENT}
 
 
-def fetch(url: str) -> dict[str, Any]:
+def fetch(url: str, profile: Profile) -> dict[str, Any]:
     """GET a URL, following redirects; return status/headers/body/timing/final-url.
 
     ``redirect`` is set (e.g. "301 Moved Permanently") only when a redirect crossed
@@ -36,16 +51,18 @@ def fetch(url: str) -> dict[str, Any]:
     is a normal response, not a connection failure, and does not trigger a retry.
     """
     try:
-        return _get(url)
+        return _get(url, profile)
     except requests.exceptions.RequestException:
         if url.lower().startswith("https://"):
-            return _get("http://" + url[len("https://"):])
+            return _get("http://" + url[len("https://"):], profile)
         raise
 
 
-def _get(url: str) -> dict[str, Any]:
+def _get(url: str, profile: Profile) -> dict[str, Any]:
     start = time.perf_counter()
-    resp = requests.get(url, headers=DEFAULT_HEADERS, timeout=TIMEOUT, allow_redirects=True)
+    # headers(url), not a cached dict: the http:// retry above must drop the client
+    # hints, which are https-only.
+    resp = requests.get(url, headers=profile.headers(url), timeout=TIMEOUT, allow_redirects=True)
     elapsed = (time.perf_counter() - start) * 1000
 
     orig = registrable_domain(urlparse(url).hostname)
@@ -76,7 +93,7 @@ def resolve_ip(domain: str) -> str | None:
 
 def get_geo(ip: str) -> dict[str, Any]:
     """Geolocate an IP via ip-api.com (free, no key)."""
-    resp = requests.get(f"http://ip-api.com/json/{ip}", headers=DEFAULT_HEADERS, timeout=TIMEOUT)
+    resp = requests.get(f"http://ip-api.com/json/{ip}", headers=API_HEADERS, timeout=TIMEOUT)
     return resp.json()
 
 

{web_scanner-2.1.3 → web_scanner-2.1.4}/webscanner/ui/activity.py

@@ -31,6 +31,7 @@ from ..colors import GREEN, RED
 from ..core.context import ScanContext
 from ..core.models import ModuleResult, ModuleStatus, ScanEvent
 from ..core.scanner import PREFETCH
+from ..net.agents import Profile
 from ..modules import all_modules
 from ..modules.dns import RECORD_TYPES
 from ..modules.links import EMPTY_EXTERNAL, EMPTY_INTERNAL
@@ -273,6 +274,31 @@ def started(target: str, count: int) -> str:
     return f"Scan: Started {_esc(target)} · {_plural(count, 'module')}."
 
 
+def agent(profile: Profile) -> str:
+    """The browser identity this scan wore.
+
+    Worth a line because the profile is chosen at random per scan: when a scan returns
+    something surprising (a block page, a tech list that shifted since last run), the
+    first question is what the site thought we were, and the answer differs run to run.
+    The label — "Chrome 151 · Windows" — is enough to reproduce it; the full UA string
+    is ~110 chars and would be ellipsised to no benefit.
+    """
+    return f"Agent: {_esc(profile.label)}."
+
+
+def cms(detected: tuple[str, str | None] | None) -> str:
+    """The Server panel's CMS row, as a log line — wording kept identical to it.
+
+    Not a module: the UI derives this from the Tech result plus the page's
+    ``<meta name="generator">``, so ``app.py`` calls this straight after the Tech
+    event and the line lands directly under Tech's.
+    """
+    if detected is None:
+        return "CMS: Not detected."
+    name, version = detected
+    return f"CMS: {_esc(f'{name} {version}' if version else name)}."
+
+
 def overall(completed: int, failed: int, total: int, seconds: float) -> str:
     """The closing line. ``seconds`` is wall-clock for the whole scan — modules run
     concurrently, so summing their durations would overstate it several-fold."""

{web_scanner-2.1.3 → web_scanner-2.1.4}/webscanner/ui/app.py

@@ -232,6 +232,7 @@ class WebScannerApp(App):
         log = self.query_one("#activity", ActivityLog)
         log.clear()
         log.add(activity.started(self.ctx.domain, len(self.modules)))
+        log.add(activity.agent(self.ctx.profile))
         self._update_progress()
 
         # Blur the input so single-key nav (←/→, q, r) reaches the app rather
@@ -282,9 +283,9 @@ class WebScannerApp(App):
             # No result guard: a failed Tech scan can still surface a CMS via the
             # page's <meta name="generator">.
             data = event.result.data if event.result is not None else None
-            self.query_one("#status-content", StatusPanel).set_ctx(
-                self.ctx, cms=_detect_cms(data, self.ctx.html)
-            )
+            detected = _detect_cms(data, self.ctx.html)
+            self.query_one("#status-content", StatusPanel).set_ctx(self.ctx, cms=detected)
+            self.query_one("#activity", ActivityLog).add(activity.cms(detected))
         self._update_progress()
 
     def on_scan_finished(self, message: ScanFinished) -> None:

{web_scanner-2.1.3 → web_scanner-2.1.4}/webscanner/ui/app.tcss

@@ -81,7 +81,7 @@ Tab.-selected {
 }
 
 #main {
-    height: 4fr;   /* 80% of the left column (activity takes the other 20%). Must
+    height: 3fr;   /* 75% of the left column (activity takes the other 25%). Must
                       stay fr, NOT 100%: that claims the whole column and silently
                       clips #activity (Vertical is overflow: hidden, so no
                       scrollbar, no error — the panel just never appears). */
@@ -140,10 +140,12 @@ Tab.-selected {
     link-background-hover: transparent;
 }
 
-/* Activity log: 20% of the left column, so it grows with the terminal (#main is
-   the other 80%). 4 rows of that go to the border + padding (box-sizing is
-   border-box), so visible lines = height - 4: ~3 at 44 rows, ~6 at 60.
-   min-height floors it at one readable line on a short terminal, where 20% would
+/* Activity log: 25% of the left column, so it grows with the terminal (#main is
+   the other 75%). 4 rows of that go to the border + padding (box-sizing is
+   border-box), so visible lines = height - 4: 5 at 44 rows, 9 at 60, 14 at 80
+   (measured, not derived — heights are whole cells, and Textual hands the rounding
+   remainder to the last child).
+   min-height floors it at one readable line on a short terminal, where 25% would
    otherwise round down to nothing but chrome.
    background: transparent overrides RichLog's $surface, which would punch an
    opaque block through the ansi_default bg. */

web_scanner-2.1.3/webscanner/core/context.py

@@ -1,41 +0,0 @@
-"""Shared per-scan context passed to every module.
-
-The orchestrator's prefetch phase populates the shared network fields (one DNS
-resolve, one HTTP GET, one TLS handshake, one geo lookup) so individual modules
-reuse them instead of refetching.
-"""
-
-from __future__ import annotations
-
-from dataclasses import dataclass, field
-from typing import Any
-
-from .. import helpers
-
-
-@dataclass
-class ScanContext:
-    # inputs
-    domain: str  # bare host, e.g. "example.com"
-    url: str  # full url, e.g. "https://example.com"
-
-    # shared prefetch results (filled by AsyncScanner.prefetch)
-    ip: str | None = None
-    html: str | None = None
-    headers: dict[str, str] = field(default_factory=dict)
-    status_code: int | None = None
-    response_time_ms: float | None = None
-    final_url: str | None = None
-    redirect_status: str | None = None  # e.g. "301 Moved Permanently" for cross-domain redirects
-    tls_cert: dict[str, Any] | None = None
-    geo: dict[str, Any] | None = None
-    fetch_error: str | None = None
-
-    @classmethod
-    def from_target(cls, target: str) -> "ScanContext":
-        domain, url = helpers.normalise(target)
-        return cls(domain=domain, url=url)
-
-    @property
-    def online(self) -> bool:
-        return self.status_code is not None and self.fetch_error is None

web_scanner-2.1.3/webscanner/modules/tech.py

@@ -1,52 +0,0 @@
-"""Technology detection via Wappalyzer.
-
-Uses ``analyze(..., scan_type="fast")`` — a pure-HTTP path (no headless browser) —
-which returns per-technology version, confidence, categories and groups. Rendered as
-one :class:`Grid` per *group* (Wappalyzer's high-level bucket, e.g. "Web servers",
-"Analytics"), stacked as titled :class:`Sections`. A technology in several groups
-appears in each of their tables.
-"""
-
-from __future__ import annotations
-
-import asyncio
-
-from wappalyzer import analyze
-
-from ..core.module import ScanModule
-from ..core.context import ScanContext
-from ..core.models import Grid, Section, Sections
-
-COLUMNS = ["Name", "Category", "Confidence", "Version"]
-#: fixed column widths so every per-group table on the tab lines up identically
-WIDTHS = [26, 22, 12, 12]
-#: bucket for technologies Wappalyzer reports without any group
-UNGROUPED = "Other"
-
-
-class TechModule(ScanModule):
-    name = "tech"
-    label = "Tech"
-
-    async def run(self, ctx: ScanContext) -> Sections:
-        def detect() -> Sections:
-            results = analyze(url=ctx.url, scan_type="fast", timeout=30)
-            techs = results.get(ctx.url) or next(iter(results.values()), {})
-            # group name -> rows; a tech lands in every group it declares
-            groups: dict[str, list[list[str]]] = {}
-            for name, info in techs.items():
-                row = [
-                    name,
-                    ", ".join(info.get("categories") or []) or "-",
-                    f"{info.get('confidence', 0)}%",
-                    info.get("version") or "-",
-                ]
-                for group in info.get("groups") or [UNGROUPED]:
-                    groups.setdefault(group, []).append(row)
-            sections = Sections()
-            for group in sorted(groups, key=str.lower):  # tables A→Z by group
-                rows = sorted(groups[group], key=lambda r: r[0].lower())  # names A→Z
-                sections.append(Section(group, Grid(COLUMNS, rows, widths=WIDTHS)))
-            return sections
-
-        return await asyncio.to_thread(detect)