web-scanner 2.1.4__tar.gz → 2.2.0__tar.gz

{web_scanner-2.1.4 → web_scanner-2.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: web-scanner
-Version: 2.1.4
+Version: 2.2.0
 Summary: An async Textual TUI for website reconnaissance — DNS, WHOIS, TLS, security, tech-stack and SEO, with no paid APIs.
 Project-URL: Homepage, https://github.com/iamramizk/web-scanner
 Project-URL: Repository, https://github.com/iamramizk/web-scanner
@@ -55,7 +55,7 @@ Description-Content-Type: text/markdown
 _An async Textual TUI for website reconnaissance — DNS, WHOIS, TLS, security, tech-stack and SEO, with no paid APIs._
 
 [![Python](https://img.shields.io/badge/Python-3.11%2B-3776AB?style=for-the-badge&logo=python&logoColor=white)](https://www.python.org/downloads/)
-[![PyPI Downloads](https://img.shields.io/pypi/dm/web-scanner?style=for-the-badge&logo=pypi&logoColor=white&color=006DAD)](https://pypi.org/project/web-scanner/)
+[![PyPI Downloads](https://img.shields.io/pepy/dt/web-scanner?style=for-the-badge&logo=pypi&logoColor=white&color=006DAD)](https://pypi.org/project/web-scanner/)
 [![Release](https://img.shields.io/github/v/release/iamramizk/web-scanner?style=for-the-badge&logo=github&color=2EA44F)](https://github.com/iamramizk/web-scanner/releases/latest)
 
 </div>
@@ -69,13 +69,14 @@ a server-status panel that stay pinned in place.
 resolvers for blocklist checks, the system `whois`, and stdlib `ssl`/`socket`. Country
 borders are embedded (Natural Earth), so even the map needs no tile service.
 
-![WebScanner](https://raw.githubusercontent.com/iamramizk/web-scanner/main/.github/screenshot-v2.1.png)
+![WebScanner — DNS tab](https://raw.githubusercontent.com/iamramizk/web-scanner/main/.github/screenshot-v2-1.png)
+
+![WebScanner — Security tab](https://raw.githubusercontent.com/iamramizk/web-scanner/main/.github/screenshot-v2-2.png)
 
 ## Contents
 
 - [Features](#features)
 - [Installation](#installation)
-  - [From source (development)](#from-source-development)
 - [Usage](#usage)
 - [Updating](#updating)
 - [Keys](#keys)
@@ -84,26 +85,41 @@ borders are embedded (Natural Earth), so even the map needs no tile service.
 
 Ten tabs, scanned concurrently and rendered live as each module finishes:
 
-- **DNS** — A / AAAA / NS / CNAME / SOA / MX / TXT / CAA / DS / DNSKEY records, plus email
-  authentication folded in: DMARC (`_dmarc`) and DKIM (probes ~40 known selectors, each hit
-  labelled with the email provider it points to, where the selector identifies one).
-- **Whois** — parsed system `whois`, with rich gTLD and ccTLD field support (registrar,
-  dates, nameservers, per-contact details).
-- **Subdomains** — discovered natively from TLS certificate SANs and `socket` probes of
-  common subdomains — no third-party enumeration services.
-- **SSL** — certificate issuer, subject, SANs, validity window, trust and expiry, parsed
-  from the live TLS handshake.
-- **Security** — TCP connect port scan, presence of HTTP security headers (CSP, HSTS,
-  X-Frame-Options, …), and blocklist status across public filtering resolvers
-  (AdGuard, CleanBrowsing, Cloudflare, Google, OpenDNS, Quad9).
+- **DNS** — records plus email authentication, folded into one tab.
+  - A / AAAA / NS / CNAME / SOA / MX / TXT / CAA / DS / DNSKEY records
+  - DMARC (`_dmarc`) and DKIM (probes ~40 selectors, each labelled with its email provider)
+  - SPF `all` qualifier, and an **Email Spoofing** verdict (Protected / Weak / Vulnerable)
+    computed from DMARC enforcement, not SPF alone
+- **Whois** — parsed system `whois`, with rich gTLD and ccTLD field support.
+  - Registrar, registration / expiry dates, nameservers
+  - Per-contact details where published
+- **Subdomains** — discovered natively, no third-party enumeration services.
+  - TLS certificate SANs
+  - `socket` probes of common subdomains
+- **SSL** — the live TLS certificate, parsed from the handshake.
+  - Issuer, subject, SANs
+  - Validity window, trust and expiry
+- **Security** — four checks, each a sub-table.
+  - **WAF detection** — passive header/cookie fingerprinting plus an active probe that
+    sends obvious attack payloads to see if it gets blocked (names Cloudflare, Sucuri,
+    Akamai, ModSecurity, …)
+  - **Open ports** — TCP connect scan of common ports
+  - **HTTP security headers** — CSP, HSTS, X-Frame-Options, … present or not
+  - **Blocklists** — status across public filtering resolvers (AdGuard, CleanBrowsing,
+    Cloudflare, Google, OpenDNS, Quad9)
 - **Headers** — the full set of HTTP response headers.
-- **Tech** — technology-stack detection via [Wappalyzer](https://github.com/tunetheweb/wappalyzer),
-  listing each technology with its category, confidence, groups and version.
-- **SEO** — page content (title/description with length hints, H1–H3, social links), top
-  keyword n-grams, `robots.txt` and sitemaps, and JSON-LD structured data.
-- **Sitemap** — the site's URLs discovered from its `sitemap.xml` (recursing into nested
-  sitemap indexes) and rebuilt into a clickable URL-path tree — no crawling.
-- **Links** — internal and external links, with their anchor text.
+- **Tech** — technology-stack detection via [Wappalyzer](https://github.com/tunetheweb/wappalyzer).
+  - Each technology with its category, confidence, groups and version
+- **SEO** — everything on-page that search engines read.
+  - Title / description with length hints, H1–H3, social links
+  - Top keyword n-grams (1 / 2 / 3-word)
+  - `robots.txt` and declared sitemaps
+  - JSON-LD structured data
+- **Sitemap** — the site's URLs as a clickable path tree, no crawling.
+  - Discovered from `sitemap.xml`, recursing into nested sitemap indexes
+  - Rebuilt into a folder tree keyed by URL path
+- **Links** — links found on the page, with their anchor text.
+  - Internal and external, split into sub-tables
 
 Alongside the tabs, three fixed panels:
 
@@ -112,7 +128,9 @@ Alongside the tabs, three fixed panels:
 - **Country map** — real country outlines auto-framed around the server's location, drawn
   with braille characters (`+` / `-` to zoom).
 - **Server** — online status and response time, the final URL after redirects, IP,
-  geolocation, ISP, AS, hosting provider and detected CMS (name and version).
+  geolocation, ISP, AS, hosting provider and detected CMS (name and version). When many
+  sites share the server's IP, it flags the address as shared and counts how many distinct
+  domains resolve to it (via a free reverse-IP lookup).
 
 Requests to the site being scanned wear a **coherent desktop-Chrome identity** — a real
 User-Agent plus the headers Chrome actually sends beside it — picked once per scan and

{web_scanner-2.1.4 → web_scanner-2.2.0}/README.md

@@ -5,7 +5,7 @@
 _An async Textual TUI for website reconnaissance — DNS, WHOIS, TLS, security, tech-stack and SEO, with no paid APIs._
 
 [![Python](https://img.shields.io/badge/Python-3.11%2B-3776AB?style=for-the-badge&logo=python&logoColor=white)](https://www.python.org/downloads/)
-[![PyPI Downloads](https://img.shields.io/pypi/dm/web-scanner?style=for-the-badge&logo=pypi&logoColor=white&color=006DAD)](https://pypi.org/project/web-scanner/)
+[![PyPI Downloads](https://img.shields.io/pepy/dt/web-scanner?style=for-the-badge&logo=pypi&logoColor=white&color=006DAD)](https://pypi.org/project/web-scanner/)
 [![Release](https://img.shields.io/github/v/release/iamramizk/web-scanner?style=for-the-badge&logo=github&color=2EA44F)](https://github.com/iamramizk/web-scanner/releases/latest)
 
 </div>
@@ -19,13 +19,14 @@ a server-status panel that stay pinned in place.
 resolvers for blocklist checks, the system `whois`, and stdlib `ssl`/`socket`. Country
 borders are embedded (Natural Earth), so even the map needs no tile service.
 
-![WebScanner](https://raw.githubusercontent.com/iamramizk/web-scanner/main/.github/screenshot-v2.1.png)
+![WebScanner — DNS tab](https://raw.githubusercontent.com/iamramizk/web-scanner/main/.github/screenshot-v2-1.png)
+
+![WebScanner — Security tab](https://raw.githubusercontent.com/iamramizk/web-scanner/main/.github/screenshot-v2-2.png)
 
 ## Contents
 
 - [Features](#features)
 - [Installation](#installation)
-  - [From source (development)](#from-source-development)
 - [Usage](#usage)
 - [Updating](#updating)
 - [Keys](#keys)
@@ -34,26 +35,41 @@ borders are embedded (Natural Earth), so even the map needs no tile service.
 
 Ten tabs, scanned concurrently and rendered live as each module finishes:
 
-- **DNS** — A / AAAA / NS / CNAME / SOA / MX / TXT / CAA / DS / DNSKEY records, plus email
-  authentication folded in: DMARC (`_dmarc`) and DKIM (probes ~40 known selectors, each hit
-  labelled with the email provider it points to, where the selector identifies one).
-- **Whois** — parsed system `whois`, with rich gTLD and ccTLD field support (registrar,
-  dates, nameservers, per-contact details).
-- **Subdomains** — discovered natively from TLS certificate SANs and `socket` probes of
-  common subdomains — no third-party enumeration services.
-- **SSL** — certificate issuer, subject, SANs, validity window, trust and expiry, parsed
-  from the live TLS handshake.
-- **Security** — TCP connect port scan, presence of HTTP security headers (CSP, HSTS,
-  X-Frame-Options, …), and blocklist status across public filtering resolvers
-  (AdGuard, CleanBrowsing, Cloudflare, Google, OpenDNS, Quad9).
+- **DNS** — records plus email authentication, folded into one tab.
+  - A / AAAA / NS / CNAME / SOA / MX / TXT / CAA / DS / DNSKEY records
+  - DMARC (`_dmarc`) and DKIM (probes ~40 selectors, each labelled with its email provider)
+  - SPF `all` qualifier, and an **Email Spoofing** verdict (Protected / Weak / Vulnerable)
+    computed from DMARC enforcement, not SPF alone
+- **Whois** — parsed system `whois`, with rich gTLD and ccTLD field support.
+  - Registrar, registration / expiry dates, nameservers
+  - Per-contact details where published
+- **Subdomains** — discovered natively, no third-party enumeration services.
+  - TLS certificate SANs
+  - `socket` probes of common subdomains
+- **SSL** — the live TLS certificate, parsed from the handshake.
+  - Issuer, subject, SANs
+  - Validity window, trust and expiry
+- **Security** — four checks, each a sub-table.
+  - **WAF detection** — passive header/cookie fingerprinting plus an active probe that
+    sends obvious attack payloads to see if it gets blocked (names Cloudflare, Sucuri,
+    Akamai, ModSecurity, …)
+  - **Open ports** — TCP connect scan of common ports
+  - **HTTP security headers** — CSP, HSTS, X-Frame-Options, … present or not
+  - **Blocklists** — status across public filtering resolvers (AdGuard, CleanBrowsing,
+    Cloudflare, Google, OpenDNS, Quad9)
 - **Headers** — the full set of HTTP response headers.
-- **Tech** — technology-stack detection via [Wappalyzer](https://github.com/tunetheweb/wappalyzer),
-  listing each technology with its category, confidence, groups and version.
-- **SEO** — page content (title/description with length hints, H1–H3, social links), top
-  keyword n-grams, `robots.txt` and sitemaps, and JSON-LD structured data.
-- **Sitemap** — the site's URLs discovered from its `sitemap.xml` (recursing into nested
-  sitemap indexes) and rebuilt into a clickable URL-path tree — no crawling.
-- **Links** — internal and external links, with their anchor text.
+- **Tech** — technology-stack detection via [Wappalyzer](https://github.com/tunetheweb/wappalyzer).
+  - Each technology with its category, confidence, groups and version
+- **SEO** — everything on-page that search engines read.
+  - Title / description with length hints, H1–H3, social links
+  - Top keyword n-grams (1 / 2 / 3-word)
+  - `robots.txt` and declared sitemaps
+  - JSON-LD structured data
+- **Sitemap** — the site's URLs as a clickable path tree, no crawling.
+  - Discovered from `sitemap.xml`, recursing into nested sitemap indexes
+  - Rebuilt into a folder tree keyed by URL path
+- **Links** — links found on the page, with their anchor text.
+  - Internal and external, split into sub-tables
 
 Alongside the tabs, three fixed panels:
 
@@ -62,7 +78,9 @@ Alongside the tabs, three fixed panels:
 - **Country map** — real country outlines auto-framed around the server's location, drawn
   with braille characters (`+` / `-` to zoom).
 - **Server** — online status and response time, the final URL after redirects, IP,
-  geolocation, ISP, AS, hosting provider and detected CMS (name and version).
+  geolocation, ISP, AS, hosting provider and detected CMS (name and version). When many
+  sites share the server's IP, it flags the address as shared and counts how many distinct
+  domains resolve to it (via a free reverse-IP lookup).
 
 Requests to the site being scanned wear a **coherent desktop-Chrome identity** — a real
 User-Agent plus the headers Chrome actually sends beside it — picked once per scan and

{web_scanner-2.1.4 → web_scanner-2.2.0}/webscanner/__init__.py

@@ -1,3 +1,3 @@
 """web-scanner v2 — async website reconnaissance TUI package."""
 
-__version__ = "2.1.4"
+__version__ = "2.2.0"

{web_scanner-2.1.4 → web_scanner-2.2.0}/webscanner/colors.py

@@ -2,6 +2,9 @@
 
 GREEN = "green"
 RED = "#FF6C64"
+#: matches the app's primary theme blue ($primary, textual-dark) — progress bar,
+#: selected-tab chip, keybar hints.
+BLUE = "#0178D4"
 #: muted value/secondary text — matches the unselected-tab colour
 #: ($text-muted = white @ 60% over the dark theme bg).
 MUTED = "#9D9D9D"

{web_scanner-2.1.4 → web_scanner-2.2.0}/webscanner/core/context.py

@@ -38,6 +38,13 @@ class ScanContext:
     geo: dict[str, Any] | None = None
     fetch_error: str | None = None
 
+    #: Whether the IP is shared by many sites (reverse-IP lookup). ``None`` until the
+    #: lookup resolves — and stays None on any error/rate-limit, so the panel shows
+    #: nothing rather than guess. ``shared_ip_count`` is the unique-domain tally shown
+    #: alongside. Filled by AsyncScanner concurrently with the modules, never prefetch.
+    ip_shared: bool | None = None
+    shared_ip_count: int | None = None
+
     @classmethod
     def from_target(cls, target: str) -> "ScanContext":
         domain, url = helpers.normalise(target)

{web_scanner-2.1.4 → web_scanner-2.2.0}/webscanner/core/scanner.py

@@ -16,7 +16,7 @@ import asyncio
 import time
 from collections.abc import Callable, Iterable
 
-from ..net import http
+from ..net import http, psl
 from .context import ScanContext
 from .models import ModuleResult, ModuleStatus, ScanEvent
 from .module import ScanModule
@@ -24,6 +24,11 @@ from .module import ScanModule
 EventCallback = Callable[[ScanEvent], None]
 
 PREFETCH = "__prefetch__"
+SHARED_IP = "__shared_ip__"
+
+#: Unique registrable domains (eTLD+1) on the IP before we call it "shared". A single
+#: company's handful of vanity domains stays under it; shared hosting / CDN IPs clear it.
+SHARED_IP_THRESHOLD = 10
 
 
 def _is_empty(data: object) -> bool:
@@ -81,6 +86,29 @@ class AsyncScanner:
 
         await asyncio.gather(_fetch(), _geo(), _tls())
 
+    async def _shared_ip(self) -> None:
+        """Flag a shared IP on the Server panel via a reverse-IP lookup.
+
+        Runs concurrently with the modules, *never* in the prefetch barrier: it's a
+        slow third-party API no module depends on, so joining prefetch would let it
+        stall all ten. Failure-isolated and best-effort — any error leaves
+        ``ip_shared`` None (the panel shows nothing). Emits ``SHARED_IP`` only on a
+        real result, so the UI refreshes the panel just once, when it lands.
+        """
+        if not self.ctx.ip:
+            return
+        try:
+            hosts = await asyncio.to_thread(http.reverse_ip_lookup, self.ctx.ip)
+            if hosts is None:
+                return
+            domains = {psl.registrable_domain(h) for h in hosts}
+            domains.discard("")
+            self.ctx.shared_ip_count = len(domains)
+            self.ctx.ip_shared = len(domains) >= SHARED_IP_THRESHOLD
+            self._emit(SHARED_IP, ModuleStatus.DONE)
+        except Exception:  # noqa: BLE001 - best effort, must never break the scan
+            pass
+
     async def _run_module(self, module: ScanModule) -> ModuleResult:
         self._emit(module.name, ModuleStatus.RUNNING)
         start = time.perf_counter()
@@ -100,5 +128,8 @@ class AsyncScanner:
         self._emit(PREFETCH, ModuleStatus.RUNNING)
         await self.prefetch()
         self._emit(PREFETCH, ModuleStatus.DONE)
-        await asyncio.gather(*(self._run_module(m) for m in self.modules))
+        await asyncio.gather(
+            self._shared_ip(),
+            *(self._run_module(m) for m in self.modules),
+        )
         return self.results

{web_scanner-2.1.4 → web_scanner-2.2.0}/webscanner/modules/dns.py

@@ -9,10 +9,12 @@ when found (no "present: yes/no" noise).
 from __future__ import annotations
 
 import asyncio
+import re
 from typing import Any
 
 import pydig
 
+from ..colors import GREEN, RED
 from ..core.module import ScanModule
 from ..core.context import ScanContext
 
@@ -63,6 +65,77 @@ def _is_dkim(records: list[str]) -> bool:
     return any(("dkim1" in r.lower() or "p=" in r.lower()) for r in records)
 
 
+# ---- email-spoofing assessment -------------------------------------------
+#
+# A *configuration* verdict from what DNS already tells us — the same thing MXToolbox /
+# dmarcian report, not a live send test. The load-bearing insight: **DMARC enforcement,
+# not SPF, is what stops visible (From-header) spoofing.** SPF only authenticates the
+# envelope/return-path, so a domain with a perfect `-all` SPF but no DMARC (or `p=none`)
+# is still spoofable in a way the recipient sees. So the verdict keys off DMARC policy,
+# with SPF only breaking the tie when DMARC is absent.
+
+_ALL_MECHANISM = re.compile(r"([~+?-])all\b", re.I)
+_DMARC_TAG = re.compile(r"\b(p|sp|pct)\s*=\s*([^;]+)", re.I)
+
+
+def _spf_qualifier(txt_records: list[str]) -> str | None:
+    """The qualifier of the SPF record's ``all`` mechanism (``-all``/``~all``/``?all``/
+    ``+all``), or ``None`` if there's no ``v=spf1`` record or it carries no ``all``."""
+    for record in txt_records:
+        if record.strip().lower().startswith("v=spf1"):
+            if match := _ALL_MECHANISM.search(record):
+                return f"{match.group(1)}all"
+            return None  # SPF present but no `all` — treat as no explicit policy
+    return None
+
+
+def _dmarc_policy(dmarc_records: list[str]) -> tuple[str | None, int]:
+    """``(policy, pct)`` from the first DMARC record — policy is ``p=`` lower-cased
+    (``reject``/``quarantine``/``none``) or ``None``; ``pct`` defaults to 100."""
+    for record in dmarc_records:
+        tags = {m.group(1).lower(): m.group(2).strip() for m in _DMARC_TAG.finditer(record)}
+        policy = tags.get("p", "").lower() or None
+        try:
+            pct = int(tags.get("pct", "100"))
+        except ValueError:
+            pct = 100
+        return policy, pct
+    return None, 100
+
+
+def assess_spoofing(
+    txt_records: list[str], dmarc_records: list[str], has_dkim: bool
+) -> tuple[str, str]:
+    """``(verdict, reason)`` — verdict is ``Protected`` / ``Weak`` / ``Vulnerable``.
+
+    Pure (no ctx, no I/O) so it's unit-testable like the activity-line catalogue.
+    """
+    spf = _spf_qualifier(txt_records)
+    policy, pct = _dmarc_policy(dmarc_records)
+
+    if policy in ("reject", "quarantine"):
+        if pct < 100:
+            return "Weak", f"DMARC p={policy} but only pct={pct}"
+        label = "Protected" if policy == "reject" else "Protected (quarantine)"
+        return label, f"DMARC p={policy}"
+    if policy == "none":
+        return "Weak", "DMARC p=none — monitoring only, mail still delivered"
+    # No enforcing DMARC — SPF alone can't stop From-header spoofing.
+    if spf == "-all":
+        return "Weak", "SPF -all but no DMARC policy"
+    if spf:
+        return "Vulnerable", f"no DMARC and SPF {spf} (not enforcing)"
+    return "Vulnerable", "no DMARC policy and no SPF record"
+
+
+def _verdict_cell(verdict: str, reason: str) -> str:
+    """Colour the verdict verb (green Protected, else red — Weak and Vulnerable are both
+    spoofable) with the reason kept alongside. A value with ``[/]`` renders via markup in
+    ``_value_cell``."""
+    colour = GREEN if verdict.startswith("Protected") else RED
+    return f"[{colour}]{verdict}[/] — {reason}"
+
+
 class DnsModule(ScanModule):
     name = "dns"
     label = "DNS"
@@ -100,4 +173,11 @@ class DnsModule(ScanModule):
                 f"{s}  ({DKIM_SELECTORS[s]})" if DKIM_SELECTORS[s] else s
                 for s in found
             ]
+
+        # Derived email-spoofing verdict from the records above (no extra network work).
+        # SPF sits inside the TXT records; surface its `all` qualifier, then the verdict.
+        if (spf := _spf_qualifier(out.get("TXT", []))) is not None:
+            out["SPF"] = [spf]
+        verdict, reason = assess_spoofing(out.get("TXT", []), dmarc_records, bool(found))
+        out["Email Spoofing"] = [_verdict_cell(verdict, reason)]
         return out