web-scanner 2.2.2__tar.gz → 2.2.3__tar.gz

{web_scanner-2.2.2 → web_scanner-2.2.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: web-scanner
-Version: 2.2.2
+Version: 2.2.3
 Summary: An async Textual TUI for website reconnaissance — DNS, WHOIS, TLS, security, tech-stack and SEO, with no paid APIs.
 Project-URL: Homepage, https://github.com/iamramizk/web-scanner
 Project-URL: Repository, https://github.com/iamramizk/web-scanner
@@ -69,9 +69,7 @@ a server-status panel that stay pinned in place.
 resolvers for blocklist checks, the system `whois`, and stdlib `ssl`/`socket`. Country
 borders are embedded (Natural Earth), so even the map needs no tile service.
 
-![WebScanner — DNS tab](https://raw.githubusercontent.com/iamramizk/web-scanner/main/.github/screenshot-v2-1.png)
-
-![WebScanner — Security tab](https://raw.githubusercontent.com/iamramizk/web-scanner/main/.github/screenshot-v2-2.png)
+![WebScanner — DNS tab, with the responsive narrow layout inset](https://raw.githubusercontent.com/iamramizk/web-scanner/main/.github/screenshot-v2-2-2.png)
 
 ## Contents
 
@@ -130,7 +128,11 @@ Alongside the tabs, three fixed panels:
 - **Server** — online status and response time, the final URL after redirects, IP,
   geolocation, ISP, AS, hosting provider and detected CMS (name and version). When many
   sites share the server's IP, it flags the address as shared and counts how many distinct
-  domains resolve to it (via a free reverse-IP lookup).
+  domains resolve to it (via a free reverse-IP lookup, [hackertarget](https://hackertarget.com/),
+  rate-limited to 50 requests/day).
+
+The layout is **responsive**: on a narrow terminal it collapses to a single full-width
+column and the map, Server and Activity panels fold into their own tabs.
 
 Requests to the site being scanned wear a **coherent desktop-Chrome identity** — a real
 User-Agent plus the headers Chrome actually sends beside it — picked once per scan and
@@ -174,11 +176,19 @@ webscan example.com
 You can also run it as a module (`python -m webscanner example.com`), or from a source
 checkout without installing (`python app.py example.com`).
 
-Press `s` inside the app to save every tab to CSV. When installed, results are written to a
-`./<domain>_<timestamp>/` folder in your current directory.
+Once a scan finishes you can stay in the app: press `r` to rescan the same domain, or `esc`
+to edit the domain and scan a different one — no need to restart.
+
+Press `s` to save every tab to CSV (plus a `server.csv` for the fixed Server panel, which
+has no tab of its own). Files go into a `<domain>_<timestamp>/` folder — under `output/`
+when running from a source checkout, or straight in your current directory when installed.
 
 ## Updating
 
+The app checks PyPI in the background once a day and, when a newer release is out, marks the
+version in the footer with an orange dot and adds a line to the Activity Log — so you'll know
+when it's worth upgrading.
+
 ```bash
 pipx upgrade web-scanner
 ```
@@ -195,6 +205,8 @@ pipx install --force git+https://github.com/iamramizk/web-scanner
 | Key              | Action                                                              |
 | ---------------- | ------------------------------------------------------------------- |
 | `←` / `→` `Tab`  | Switch tabs                                                         |
+| `1`–`9` `0`      | Jump straight to the Nth tab (`0` = 10th)                           |
+| `PgUp` / `PgDn`  | Scroll the main panel up / down                                     |
 | `+` / `-`        | Zoom the country map in / out                                       |
 | `↑` / `↓` `enter`| Navigate the Sitemap tree (`space` expands/collapses all)          |
 | `r`              | Rescan                                                              |

{web_scanner-2.2.2 → web_scanner-2.2.3}/README.md

@@ -19,9 +19,7 @@ a server-status panel that stay pinned in place.
 resolvers for blocklist checks, the system `whois`, and stdlib `ssl`/`socket`. Country
 borders are embedded (Natural Earth), so even the map needs no tile service.
 
-![WebScanner — DNS tab](https://raw.githubusercontent.com/iamramizk/web-scanner/main/.github/screenshot-v2-1.png)
-
-![WebScanner — Security tab](https://raw.githubusercontent.com/iamramizk/web-scanner/main/.github/screenshot-v2-2.png)
+![WebScanner — DNS tab, with the responsive narrow layout inset](https://raw.githubusercontent.com/iamramizk/web-scanner/main/.github/screenshot-v2-2-2.png)
 
 ## Contents
 
@@ -80,7 +78,11 @@ Alongside the tabs, three fixed panels:
 - **Server** — online status and response time, the final URL after redirects, IP,
   geolocation, ISP, AS, hosting provider and detected CMS (name and version). When many
   sites share the server's IP, it flags the address as shared and counts how many distinct
-  domains resolve to it (via a free reverse-IP lookup).
+  domains resolve to it (via a free reverse-IP lookup, [hackertarget](https://hackertarget.com/),
+  rate-limited to 50 requests/day).
+
+The layout is **responsive**: on a narrow terminal it collapses to a single full-width
+column and the map, Server and Activity panels fold into their own tabs.
 
 Requests to the site being scanned wear a **coherent desktop-Chrome identity** — a real
 User-Agent plus the headers Chrome actually sends beside it — picked once per scan and
@@ -124,11 +126,19 @@ webscan example.com
 You can also run it as a module (`python -m webscanner example.com`), or from a source
 checkout without installing (`python app.py example.com`).
 
-Press `s` inside the app to save every tab to CSV. When installed, results are written to a
-`./<domain>_<timestamp>/` folder in your current directory.
+Once a scan finishes you can stay in the app: press `r` to rescan the same domain, or `esc`
+to edit the domain and scan a different one — no need to restart.
+
+Press `s` to save every tab to CSV (plus a `server.csv` for the fixed Server panel, which
+has no tab of its own). Files go into a `<domain>_<timestamp>/` folder — under `output/`
+when running from a source checkout, or straight in your current directory when installed.
 
 ## Updating
 
+The app checks PyPI in the background once a day and, when a newer release is out, marks the
+version in the footer with an orange dot and adds a line to the Activity Log — so you'll know
+when it's worth upgrading.
+
 ```bash
 pipx upgrade web-scanner
 ```
@@ -145,6 +155,8 @@ pipx install --force git+https://github.com/iamramizk/web-scanner
 | Key              | Action                                                              |
 | ---------------- | ------------------------------------------------------------------- |
 | `←` / `→` `Tab`  | Switch tabs                                                         |
+| `1`–`9` `0`      | Jump straight to the Nth tab (`0` = 10th)                           |
+| `PgUp` / `PgDn`  | Scroll the main panel up / down                                     |
 | `+` / `-`        | Zoom the country map in / out                                       |
 | `↑` / `↓` `enter`| Navigate the Sitemap tree (`space` expands/collapses all)          |
 | `r`              | Rescan                                                              |

{web_scanner-2.2.2 → web_scanner-2.2.3}/webscanner/__init__.py

@@ -1,3 +1,3 @@
 """web-scanner v2 — async website reconnaissance TUI package."""
 
-__version__ = "2.2.2"
+__version__ = "2.2.3"

{web_scanner-2.2.2 → web_scanner-2.2.3}/webscanner/modules/dns.py

@@ -151,6 +151,12 @@ class DnsModule(ScanModule):
                     out[rtype] = res
             return out
 
+        async def www_cname() -> list[str]:
+            # The apex almost never has a CNAME (forbidden alongside its SOA/NS), so
+            # the standard CNAME lookup above is usually empty. Hosting providers put
+            # the CNAME on `www.` instead — surface it when the apex has none.
+            return await asyncio.to_thread(pydig.query, f"www.{domain}", "CNAME")
+
         async def dmarc() -> list[str]:
             txt = await asyncio.to_thread(pydig.query, f"_dmarc.{domain}", "TXT")
             return [t for t in txt if "dmarc1" in t.lower()]
@@ -159,12 +165,17 @@ class DnsModule(ScanModule):
             txt = await asyncio.to_thread(pydig.query, f"{sel}._domainkey.{domain}", "TXT")
             return sel if txt and _is_dkim(txt) else None
 
-        out, dmarc_records, dkim_hits = await asyncio.gather(
+        out, www, dmarc_records, dkim_hits = await asyncio.gather(
             asyncio.to_thread(records),
+            www_cname(),
             dmarc(),
             asyncio.gather(*(dkim_selector(s) for s in DKIM_SELECTORS)),
         )
 
+        # Show the www CNAME when the apex has none (the common hosting-provider case).
+        if www and "CNAME" not in out:
+            out["CNAME (www)"] = www
+
         if dmarc_records:
             out["DMARC"] = dmarc_records
         found = [s for s in dkim_hits if s]

{web_scanner-2.2.2 → web_scanner-2.2.3}/webscanner/ui/activity.py

@@ -29,7 +29,7 @@ from urllib.parse import urlparse
 
 from rich.markup import escape
 
-from ..colors import GREEN, RED
+from ..colors import BLUE, GREEN, RED
 from ..net.psl import registrable_domain
 from ..core.context import ScanContext
 from ..core.models import ModuleResult, ModuleStatus, ScanEvent
@@ -364,6 +364,13 @@ def waf(result: ModuleResult) -> str | None:
     return f"{line} · actively blocking." if blocked else f"{line}."
 
 
+def update_available(latest: str) -> str:
+    """A newer release exists on PyPI. Emitted once, after the scan's own closing
+    line, only when a newer version was actually found — see net/version_check.py.
+    Complements (does not replace) the status-bar version dot."""
+    return f"Update: [{BLUE}]v{_esc(latest)} available[/] — run `pipx upgrade web-scanner`."
+
+
 def overall(completed: int, failed: int, total: int, seconds: float) -> str:
     """The closing line. ``seconds`` is wall-clock for the whole scan — modules run
     concurrently, so summing their durations would overstate it several-fold."""

{web_scanner-2.2.2 → web_scanner-2.2.3}/webscanner/ui/app.py

@@ -25,7 +25,7 @@ from textual.message import Message
 from textual.widgets import Input, LoadingIndicator, Static
 
 from .. import __version__
-from ..colors import BLUE, GREEN, ORANGE
+from ..colors import ORANGE
 from ..core import AsyncScanner, ModuleStatus, ScanContext, ScanEvent
 from ..core.scanner import PREFETCH, SHARED_IP
 from ..modules import all_modules
@@ -39,10 +39,10 @@ from .widgets import ActivityLog, MapPanel, SitemapTree, StatusPanel, TabBar, Ta
 _BAR_WIDTH = 22
 _BAR_DIM = "grey30"
 
-#: colour of the status-bar version dot per update state (see net/version_check.py):
-#: blue until the background check answers / if it couldn't, green = up to date,
-#: orange = a newer PyPI release is available.
-_DOT_COLOURS = {"unknown": BLUE, "latest": GREEN, "outdated": ORANGE}
+#: The status-bar version dot is shown only when a newer PyPI release is available
+#: (`_version_status == "outdated"`; see net/version_check.py) — an orange `•`. The
+#: initial/unknown and up-to-date states show no dot, keeping the footer quiet until
+#: there's something to say.
 
 #: below this terminal width the layout collapses to a single column (the fixed map
 #: + Server panels become tabs, activity becomes a tab) — see _apply_narrow.
@@ -56,19 +56,29 @@ _PSEUDO_NAMES = frozenset(name for name, _ in _PSEUDO_TABS)
 _PSEUDO_LABELS = {name: label for name, label in _PSEUDO_TABS}
 
 
+#: Wappalyzer categories that name the CMS, in priority order. ``CMS`` is the reliable
+#: signal (the broader "Content" *group* also covers non-CMS tools). ``Page builders``
+#: is a fallback for standalone site builders that carry no ``CMS`` tag (Webflow, Wix,
+#: Squarespace, Duda, Framer). It is checked *only after* ``CMS`` comes up empty, because
+#: the same category also holds WordPress *plugins* that are not the CMS (Elementor, Divi,
+#: WPBakery) — on those sites Wappalyzer tags WordPress ``CMS`` and the first pass wins.
+_CMS_CATEGORIES = ("CMS", "Page builders")
+
+
 def _cms_from_tech(data: object) -> tuple[str, str | None] | None:
     """The CMS (name, version) from the Tech result, or ``None`` if none detected.
 
     The Tech result is a ``Sections`` of per-group ``Grid``s whose rows are
-    ``[name, categories, confidence, version]`` (see ``modules/tech.py``). A CMS is
-    any tech Wappalyzer tags with the ``CMS`` category — the reliable signal (the
-    broader "Content" *group* also covers non-CMS tools). ``categories`` is a
-    ", "-joined string; version is ``"-"`` when unknown → returned as ``None``.
+    ``[name, categories, confidence, version]`` (see ``modules/tech.py``). A CMS is any
+    tech Wappalyzer tags with a category in ``_CMS_CATEGORIES`` — tried in order, so a
+    real ``CMS`` hit always beats a ``Page builders`` one across the whole result.
+    ``categories`` is a ", "-joined string; version is ``"-"`` when unknown → ``None``.
     """
-    for section in data or []:
-        for name, categories, _confidence, version in section.data:
-            if "CMS" in [c.strip() for c in str(categories).split(",")]:
-                return name, (version if version and version != "-" else None)
+    for target in _CMS_CATEGORIES:
+        for section in data or []:
+            for name, categories, _confidence, version in section.data:
+                if target in [c.strip() for c in str(categories).split(",")]:
+                    return name, (version if version and version != "-" else None)
     return None
 
 
@@ -117,12 +127,12 @@ def _same_cms(left: str, right: str) -> bool:
 def _detect_cms(tech_data: object, html: str | None) -> tuple[str, str | None] | None:
     """The CMS (name, version) for the Server panel, or ``None`` if nothing detected.
 
-    Wappalyzer's ``CMS`` category is the primary signal — it's curated, so it won't
-    mistake a page builder or analytics tag for the CMS. The ``<meta name="generator">``
-    tag covers the two cases it misses: a CMS Wappalyzer has no fingerprint for (e.g.
-    Sitefinity, Webflow) → the generator is used outright; and a CMS it detects but
-    can't version → the generator supplies the version, but only when it names the
-    *same* product, so an "Elementor 3.x" generator can't hijack a "WordPress" hit.
+    ``_cms_from_tech`` is the primary signal (Wappalyzer's ``CMS`` category, then a
+    ``Page builders`` fallback — see there). The ``<meta name="generator">`` tag covers
+    the two cases it misses: a CMS Wappalyzer has no fingerprint for at all (e.g.
+    Sitefinity) → the generator is used outright; and a CMS it detects but can't version
+    → the generator supplies the version, but only when it names the *same* product, so
+    an "Elementor 3.x" generator can't hijack a "WordPress" hit.
     """
     tech = _cms_from_tech(tech_data)
     generators = _generators(html)
@@ -169,11 +179,13 @@ class VersionChecked(Message):
     """Result of the background PyPI update check (see net/version_check.py).
 
     ``status`` is one of ``"unknown"`` / ``"latest"`` / ``"outdated"`` — it drives
-    the colour of the status-bar version dot.
+    the colour of the status-bar version dot. ``latest`` is the newer version string
+    when ``status == "outdated"`` (else None) — used for the Activity Log line.
     """
 
-    def __init__(self, status: str) -> None:
+    def __init__(self, status: str, latest: str | None) -> None:
         self.status = status
+        self.latest = latest
         super().__init__()
 
 
@@ -239,6 +251,12 @@ class WebScannerApp(App):
         # background PyPI update check (see _check_version) — drives the colour of the
         # status-bar version dot. "unknown" (blue) until the check answers.
         self._version_status = "unknown"
+        # …and, when a newer release exists, an Activity Log line announced once after
+        # whichever finishes last: the scan or the check itself (the dot is not enough
+        # on its own — the log line spells out the upgrade command).
+        self._update_available: str | None = None
+        self._version_checked = False
+        self._update_announced = False
 
     # ---- layout -----------------------------------------------------------
 
@@ -313,20 +331,43 @@ class WebScannerApp(App):
         self._set_keybar(editing=editing)
 
     async def _check_version(self) -> None:
-        status, _latest = await asyncio.to_thread(update_status, __version__)
-        self.post_message(VersionChecked(status))
+        status, latest = await asyncio.to_thread(update_status, __version__)
+        self.post_message(VersionChecked(status, latest))
 
     def on_version_checked(self, message: VersionChecked) -> None:
         self._version_status = message.status
+        self._update_available = message.latest
+        self._version_checked = True
         self._refresh_version()
+        self._maybe_announce_update()
+
+    def _maybe_announce_update(self) -> None:
+        """Append the "update available" Activity Log line once, after the scan's own
+        closing line — so it lands as the last message. Gated on the scan being done
+        (otherwise it would jump ahead of the module lines) and on a newer release
+        actually existing; the check may finish before or after the scan, so both
+        on_version_checked and on_scan_finished call this."""
+        if (
+            self._update_announced
+            or self._scanning
+            or not self._version_checked
+            or self._update_available is None
+        ):
+            return
+        self._update_announced = True
+        self.query_one("#activity", ActivityLog).add(
+            activity.update_available(self._update_available)
+        )
 
     def _refresh_version(self) -> None:
         """Draw the version + update dot at the right of the footer, unless a scan is
         in progress (then the progress bar owns that slot — restored on finish)."""
         if self._scanning:
             return
-        dot = _DOT_COLOURS.get(self._version_status, BLUE)
-        self.query_one("#progress", Static).update(f"v{__version__} [{dot}]•[/]")
+        # Only show the dot when an update is available (hide the initial blue /
+        # up-to-date green states) — a quiet footer until there's something to say.
+        dot = f" [{ORANGE}]•[/]" if self._version_status == "outdated" else ""
+        self.query_one("#progress", Static).update(f"v{__version__}{dot}")
 
     # ---- scanning ---------------------------------------------------------
 
@@ -435,12 +476,13 @@ class WebScannerApp(App):
             )
         )
         self._refresh_version()
+        self._maybe_announce_update()
 
     # ---- progress line ----------------------------------------------------
 
     def _update_progress(self) -> None:
         """Draw a determinate progress bar (blue = done, grey = remaining) with the
-        percentage and n/total count in white, right-aligned on the footer row."""
+        percentage in white, right-aligned on the footer row."""
         if not self._scanning:
             return
         total = len(self.modules)
@@ -449,7 +491,7 @@ class WebScannerApp(App):
         blue = self.current_theme.primary
         bar = f"[{blue}]{'━' * filled}[/][{_BAR_DIM}]{'━' * (_BAR_WIDTH - filled)}[/]"
         self.query_one("#progress", Static).update(
-            f"{bar} [white]{round(frac * 100)}% {self.completed}/{total}[/]"
+            f"{bar} [white]{round(frac * 100)}%[/]"
         )
 
     # ---- tab selection ----------------------------------------------------
@@ -670,11 +712,19 @@ class WebScannerApp(App):
     def action_zoom_out(self) -> None:
         self.query_one("#map", MapPanel).zoom_by(-1)
 
+    def _page_target(self):
+        """The widget page-up/down should scroll. Usually the main panel, but the
+        Activity pseudo-tab (narrow only) shows the #activity RichLog full-height in
+        #main's place, so page keys must scroll that instead — #main is hidden there."""
+        if self._narrow and self.selected == "activity":
+            return self.query_one("#activity", ActivityLog)
+        return self.query_one("#main", VerticalScroll)
+
     def action_scroll_main_up(self) -> None:
-        self.query_one("#main", VerticalScroll).scroll_page_up()
+        self._page_target().scroll_page_up()
 
     def action_scroll_main_down(self) -> None:
-        self.query_one("#main", VerticalScroll).scroll_page_down()
+        self._page_target().scroll_page_down()
 
     def on_input_submitted(self, message: Input.Submitted) -> None:
         target = message.value.strip()
@@ -702,6 +752,10 @@ class WebScannerApp(App):
             # Compact set — the full labels overflow a phone-width footer.
             pairs = [("←/→", "Tab"), ("r", "Scan"), ("s", "Save"), ("q", "Quit")]
         else:
-            pairs = [("q", "Quit"), ("←/→", "Tab"), ("r", "Rescan"), ("s", "Save"), ("esc", "Edit domain")]
+            # PgUp/Dn scrolls #main — only shown wide; the narrow footer has no room.
+            pairs = [
+                ("q", "Quit"), ("←/→", "Tab"), ("Pg↑/↓", "Scroll"),
+                ("r", "Rescan"), ("s", "Save"), ("esc", "Edit domain"),
+            ]
         text = "   ".join(f"[b {c}]{k}[/] {label}" for k, label in pairs)
         self.query_one("#keybar", Static).update(text)

{web_scanner-2.2.2 → web_scanner-2.2.3}/webscanner/ui/tables.py

@@ -37,8 +37,9 @@ TAB_HEADERS: dict[str, tuple[str, str]] = {
 MAX_KEY_WIDTH = 34
 
 # Tabs whose key column is smart-cased (Title Case, acronyms kept upper) instead of
-# ALL CAPS. dns stays "upper" — its keys are record types (AAAA/CNAME/DNSKEY/DMARC).
-_SMART_LABEL_TABS = {"whois", "ssl", "headers"}
+# ALL CAPS. dns is smart too — its keys are mostly record-type acronyms (kept upper via
+# _ACRONYMS), but derived rows read better Title-cased ("Email Spoofing", "CNAME (www)").
+_SMART_LABEL_TABS = {"dns", "whois", "ssl", "headers"}
 
 # Tokens that render upper-case rather than Title-cased. Lowercase keys. Only needs
 # the acronyms that actually appear as key tokens in the smart-cased tabs — NOT an
@@ -49,6 +50,8 @@ _ACRONYMS: frozenset[str] = frozenset({
     "id", "url", "iana", "whois", "dnssec",
     # ssl
     "cn", "san", "ssl", "tls",
+    # dns record types (keep upper on the smart-cased dns tab)
+    "a", "aaaa", "ns", "cname", "soa", "mx", "txt", "caa", "ds", "dnskey",
     # headers / general web acronyms
     "xss", "csp", "hsts", "cors", "www", "ua", "md5", "ip", "dns",
     "http", "https", "uri", "api", "spf", "dkim", "dmarc",