@nurkamol/seo-audit 1.38.2 → 1.40.0

package/README.md

@@ -484,6 +484,85 @@ Rules with a `*` or a `:placeholder` match a shape rather than a URL, so asking
 for them literally proves nothing. They are counted and reported, never guessed
 at. A rule that works in one hop reports nothing at all.
 
+### The rest of the domain
+
+Every check above audits the site it was pointed at. `--hosts` asks a different
+question: what *else* is on this domain, and is any of it damaging the site that
+was.
+
+```bash
+node bin/seo-audit.mjs https://example.com --hosts
+```
+
+Discovery is [certificate transparency](https://certificate.transparency.dev) —
+the public log of every certificate ever issued, which is the only free, keyless
+source of hostnames there is. Two logs are asked, certspotter first and crt.sh
+second, because one is not dependable enough: asked five times in forty seconds,
+crt.sh answered twice. Verification is DNS and this tool's own fetcher, and that
+split is the whole design. The log is a *terrible* list of live hosts:
+it holds 3,425 distinct names for one large domain and roughly three thousand of
+them stopped existing years ago. So **nothing from the log is ever reported**. It
+produces candidates; a lookup and a request decide which of them are facts.
+
+Three things come out of it, and all three are invisible to a crawl of the site
+itself:
+
+- a **staging copy** nobody remembered to close — indexable, competing with
+  production for its own results, publishing whatever is being tested;
+- a **subdomain whose CNAME points at a service that is gone**, which anybody
+  can claim and then serve from a host on your domain;
+- a **second host serving the same site**, splitting every signal it earns.
+
+Plus an inventory, printed whether or not anything was wrong with it, because
+"what else is on this domain" is worth an answer on a healthy domain too:
+
+```
+  ── Hosts on example.com ──────────────────────────────────────
+  214 hostnames in certificate transparency, 9 of them resolving, 174 not looked up
+
+  ✗ staging.example.com   198.51.100.9    200  Example — Staging
+  ✗ blog.example.com                      CNAME → old-tenant.wpengine.net (gone)
+  · shop.example.com      198.51.100.20   200  Example Shop
+  · www.example.com       198.51.100.1    301 → the canonical host
+  · preview.example.com   198.51.100.31   200, noindex
+
+  Nameservers  ns1.cloudflare.com, ns2.cloudflare.com
+  Mail         aspmx.l.google.com
+  Policies     v=spf1 include:_spf.google.com ~all
+```
+
+It is off by default and stays that way. These logs are free, unauthenticated
+and rate-limited by IP, and gov.uk's 4,674 hostnames took 89 seconds to sweep —
+a crawl should not quietly spend that. When no log answers, the report says
+`hosts-not-checked` rather than showing a domain with nothing on it, for the same
+reason `tls-not-checked` exists: a missing finding reads exactly like a passing
+one. Across ten real domains that note fired twice, so it is not a rare branch.
+
+The staging check is also silent on any host that does not answer for **its own
+root**. Two real domains taught that one, in different disguises:
+`dev.gtm.github.com` answers 307 to `/login`, and that login page answers 200
+with HTML and no `noindex`; `dev.jquery.com` 301s to `bugs.jquery.com`, a
+different sibling entirely. In both cases every other clause passed and a host
+serving nothing was reported as a leaked copy of the site. One condition covers
+both, and it needs no vocabulary of login paths — which would only ever be the
+paths somebody thought of. The trade is a staging site redirecting `/` to `/en/`
+that goes unreported, which is the right way round to be wrong.
+
+It is **on by default in the macOS window and the Raycast extension, and off by
+default on the command line**. That is deliberate rather than an inconsistency:
+those two are watched by a person, where a few seconds buys a finding a crawl
+cannot otherwise see, and `npx` is a build step, where the same seconds are spent
+unasked and a third party gets called that nobody chose to call.
+
+The inventory travels with every format the engine writes — terminal, Markdown,
+HTML, JSON, the baseline, the macOS window and its PDF export. In CSV it arrives
+as rows at level `host`, the same way passing and not-checked rows already do,
+because a CSV holding two shapes is a CSV nothing can read in one go.
+
+The window reaches it from **Settings → Crawl**, and the Raycast extension from
+its preferences. The hosted deployment leaves it off unless `ALLOW_HOSTS` is set,
+because there a stranger would be spending one shared address's allowance.
+
 ### Sites without a sitemap
 
 If no sitemap can be found, the crawl follows links from the homepage instead
@@ -536,6 +615,7 @@ reason.
 | `--reports [date]` | — | List the runs kept on this machine and stop. With a date, only those finished on or after it |
 | `--since <date>` | — | Crawl only URLs the sitemap says changed on or after this date. Refuses when `lastmod` cannot answer it |
 | `--exclude <glob>` | — | Leave URLs out of the crawl. Repeatable; `*` stops at a slash, `**` does not |
+| `--hosts` | — | Also audit the rest of the domain: what other hosts exist, which resolve, and what they serve. Off by default — one slow third-party lookup plus one per candidate host |
 | `--dry-run` | — | Say what would be crawled and stop. A handful of requests instead of hundreds |
 | `--write-sitemap <file>` | — | Write the sitemap this site should have had. Refuses on a crawl that did not see the whole site |
 | `--write-llms <file>` | — | Write the `llms.txt` this site should have had, in the [llmstxt.org](https://llmstxt.org) format, from the site's own titles and descriptions. Nothing is generated or rewritten. Refuses on a partial crawl, for the same reason |
@@ -861,6 +941,10 @@ Findings come at three levels: **error** (wrong, and costing traffic), **warning
 | The link and image sweeps say so when they stop at their cap rather than implying they checked everything | note |
 | Every `og:image` actually loads, and isn't too heavy to scrape | error / warning |
 | A declared `twitter:image` loads, when it differs from `og:image` — its *absence* is fine, X falls back to Open Graph | error |
+| **A subdomain pointing at a service that is gone**, with `--hosts` — a CNAME whose target does not resolve. Anybody can register that name at the provider and serve from a host on your domain | error |
+| **A staging copy open to the index**, with `--hosts` — a host whose leftmost label names an environment (`staging`, `dev`, `uat`, `qa`, `preview`, `sandbox`…), answering 200 with HTML. Silent on one that says `noindex`, disallows crawling in its own robots.txt, canonicalises to production, or redirects there. `beta.` and `demo.` are deliberately not environments: companies ship both | warning |
+| **A second host serving the same site again**, with `--hosts` — bodies compared, not titles, and silent when its canonical points back at the canonical host | warning |
+| The host sweep says when the certificate log did not answer, and when it stopped at its cap — never silence in either case | note |
 
 ---
 

package/action.yml

@@ -47,6 +47,15 @@ inputs:
       people's servers rate-limit and bot-block; only a 404, a 410 or no answer
       at all is ever reported. Set to any non-empty value.
     required: false
+  hosts:
+    description: >
+      Also audit the rest of the domain. Asks certificate transparency what
+      other hosts exist, resolves them, and fetches the ones that are live —
+      a staging copy open to the index, a subdomain whose CNAME points at a
+      service that is gone, a second host serving the same site. Off by default
+      because it is one slow third-party lookup plus one per candidate host.
+      Set to any non-empty value.
+    required: false
   verbose:
     description: >
       Log every request as it happens. Useful when a run is slow or a site
@@ -158,6 +167,7 @@ runs:
         INPUT_PSI_SAMPLE: ${{ inputs.psi-sample }}
         INPUT_VERBOSE: ${{ inputs.verbose }}
         INPUT_CHECK_EXTERNAL: ${{ inputs.check-external }}
+        INPUT_HOSTS: ${{ inputs.hosts }}
         INPUT_SETTLE: ${{ inputs.settle }}
         INPUT_BROWSER: ${{ inputs.browser }}
         INPUT_OS: ${{ inputs.os }}
@@ -175,6 +185,7 @@ runs:
         [ -n "$INPUT_SEARCH_CONSOLE" ] && args+=(--search-console "$INPUT_SEARCH_CONSOLE")
         [ -n "$INPUT_VERBOSE" ]  && args+=(--verbose)
         [ -n "$INPUT_CHECK_EXTERNAL" ] && args+=(--check-external)
+        [ -n "$INPUT_HOSTS" ]    && args+=(--hosts)
         [ -n "$INPUT_SETTLE" ]   && args+=(--settle "$INPUT_SETTLE")
         [ -n "$INPUT_BROWSER" ]  && args+=(--browser "$INPUT_BROWSER")
         [ -n "$INPUT_OS" ]       && args+=(--os "$INPUT_OS")

package/bin/seo-audit.mjs

@@ -81,6 +81,14 @@ const HELP = `
     --check-external   also check links pointing off the site. Off by default:
                        other people's servers rate-limit and bot-block, so only
                        a 404, a 410 or no answer at all is ever reported
+    --hosts            also audit the rest of the domain. Asks certificate
+                       transparency what other hosts exist, resolves them and
+                       fetches the ones that are live: a staging copy open to
+                       the index, a subdomain whose CNAME points at a service
+                       that is gone, a second host serving the same site.
+                       Off by default here and on by default in the window and
+                       the Raycast extension — those are watched, this is a
+                       build step, and it calls a third party either way
     --no-open          with --serve, do not open a browser. It opens one when
                        a person ran the command and never when something else
                        did, so this is only for the person who wants neither
@@ -180,6 +188,7 @@ function parseArgs(argv) {
     else if (arg === '--sitemap') opts.sitemap = value();
     else if (arg === '--redirects') opts.redirects = value();
     else if (arg === '--check-external') opts.checkExternal = true;
+    else if (arg === '--hosts') opts.hosts = true;
     else if (arg === '--user-agent') opts.userAgent = value();
     else if (arg === '--serve') {
       // The port is optional: --serve on its own, or --serve 8080.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@nurkamol/seo-audit",
-  "version": "1.38.2",
+  "version": "1.40.0",
   "description": "Crawl a site's sitemap and check every page for SEO, metadata and structured-data problems that single-page graders miss. Zero dependencies.",
   "type": "module",
   "bin": {

package/src/areas.mjs

@@ -127,6 +127,9 @@ const CATEGORY_OF = {
   // The note the hosted Worker leaves where the certificate checks would
   // have been. It belongs beside them, not in Other.
   'tls-expiring': 'Site & security',
+  'subdomain-takeover': 'Site & security', 'staging-indexable': 'Site & security',
+  'duplicate-host': 'Site & security', 'hosts-not-checked': 'Site & security',
+  'host-sweep-capped': 'Site & security',
   'tls-expired': 'Site & security', 'url-uppercase': 'Site & security',
   'url-underscore': 'Site & security', 'url-space': 'Site & security',
   'header-strict-transport-security': 'Site & security',

package/src/audit.mjs

@@ -4,7 +4,7 @@ import { bodyKind, parseHtml, parseSitemap } from './parse.mjs';
 import { parseRobots, robotsVerdict } from './robots.mjs';
 import { redirectChecks } from './redirects.mjs';
 import { pageChecks, crossPageChecks, sitemapChecks } from './checks.mjs';
-import { certificateExpiry, siteChecks } from './site.mjs';
+import { certificateExpiry, siteChecks, hostChecks } from './site.mjs';
 import { linkGraph } from './graph.mjs';
 import { compareAgents } from './compare.mjs';
 import { scoreRun } from './score.mjs';
@@ -482,6 +482,18 @@ export async function audit(target, opts = {}) {
     ...(await siteChecks(origin, fetcher, pages, { ...opts, sitemapUrls: urls, bySitemap })),
   );
 
+  // What else is on this domain — a leaked staging copy, a subdomain whose
+  // CNAME points at a service that is gone, a second host serving the same
+  // site. Only when asked: it is a slow third-party lookup plus one per
+  // candidate host, and a crawl should not quietly spend that.
+  let hosts = null;
+  if (opts.hosts) {
+    onProgress?.({ phase: 'hosts', detail: 'asking certificate transparency what else is on this domain' });
+    const swept = await hostChecks(origin, fetcher, { ...opts, onProgress });
+    findings.push(...swept.findings);
+    hosts = swept.hosts;
+  }
+
   // A migration's redirect map, checked against the live site. Only when one
   // is handed over: there is nothing to infer here, and guessing at old URLs
   // would invent findings.
@@ -669,6 +681,11 @@ export async function audit(target, opts = {}) {
     redirects: Boolean(opts.redirects),
     external: Boolean(opts.checkExternal),
     compareAs: Boolean(opts.compareAs),
+    // Asked for and answered are different things. A run with --hosts whose
+    // certificate transparency lookup failed has not checked these, and
+    // `hosts-not-checked` says so in the report; scoring them as passed would
+    // hand out credit for a lookup that never happened.
+    hosts: Boolean(opts.hosts) && hosts !== null,
   };
 
   // The llms.txt this site should have had, from the same crawl and by the
@@ -704,6 +721,10 @@ export async function audit(target, opts = {}) {
       ms: Date.now() - started,
       date: new Date().toISOString().slice(0, 10),
       sitemap: source,
+      // The inventory behind the host findings. Absent unless --hosts asked
+      // for it, so a report that does not mention other hosts is one that was
+      // never asked to look rather than one that looked and found none.
+      ...(hosts ? { hosts } : {}),
     },
   };
 }

package/src/dns.mjs

@@ -0,0 +1,215 @@
+// DNS, asked over HTTPS, because that is the only resolver every runtime this
+// engine runs in actually has.
+//
+// `node:dns` is the obvious choice and it is the wrong one. The Workers runtime
+// has no such module, so anything built on it would quietly vanish from every
+// hosted report — and a missing finding reads exactly like a passing one, which
+// is the failure this project exists to refuse. `tls-expiring` already pays
+// that price and has to apologise for it in the report with `tls-not-checked`.
+// DNS does not have to: DoH is a `fetch` and a JSON body, so the CLI, the
+// Worker and both desktop shells ask the same resolver the same question.
+//
+// Cloudflare's resolver rather than Google's, and not as a preference:
+// dns.google refused a plain fetch outright while cloudflare-dns.com answered
+// the same query, and a lookup that works everywhere beats one that works in a
+// datacentre. `resolver` is a parameter so a network that blocks it can point
+// somewhere else, and so the tests never touch the internet.
+
+/** Cloudflare's DoH endpoint, in its JSON dialect. */
+export const RESOLVER = 'https://cloudflare-dns.com/dns-query';
+
+/** The certificate transparency searches this asks, in the order it asks them.
+ *
+ *  Ordered by measured reliability, not by preference. crt.sh is the one
+ *  everybody names and it is not dependable enough to be the only one: asked
+ *  five times in forty seconds it answered twice — one 502, two successes at
+ *  6.9s and 20.9s, and two timeouts at thirty. certspotter answered the same
+ *  question in 1.8 seconds. Both are free and neither needs a key, so there is
+ *  no reason to depend on one.
+ *
+ *  First answer wins rather than merging the two. The names feed a sweep that
+ *  is capped at forty anyway, so a second source would double the latency to
+ *  refine a list that gets truncated — and the whole point of the ordering is
+ *  that the fast one usually answers.
+ */
+export const CERTIFICATE_SOURCES = [
+  {
+    name: 'certspotter',
+    url: (domain) =>
+      'https://api.certspotter.com/v1/issuances' +
+      `?domain=${encodeURIComponent(domain)}&include_subdomains=true&expand=dns_names`,
+    // One issuance, many names, already an array.
+    names: (rows) => (Array.isArray(rows) ? rows.flatMap((row) => row?.dns_names ?? []) : null),
+  },
+  {
+    name: 'crt.sh',
+    url: (domain) => `https://crt.sh/?q=${encodeURIComponent(`%.${domain}`)}&output=json`,
+    // One certificate covers many names, and crt.sh packs them into one
+    // newline-separated string rather than an array.
+    names: (rows) =>
+      Array.isArray(rows) ? rows.flatMap((row) => String(row?.name_value ?? '').split('\n')) : null,
+  },
+];
+
+/** One source, asked. `null` means it did not answer. */
+async function askLog(source, domain, { timeout, fetchImpl }) {
+  const controller = new AbortController();
+  const timer = setTimeout(() => controller.abort(), timeout);
+  try {
+    const res = await fetchImpl(source.url(domain), {
+      headers: { accept: 'application/json' },
+      signal: controller.signal,
+    });
+    if (!res.ok) return null;
+    return source.names(await res.json());
+  } catch {
+    return null;
+  } finally {
+    clearTimeout(timer);
+  }
+}
+
+/** The record types this asks for, by their wire numbers, because that is what
+ *  a DoH answer is labelled with. */
+const TYPE = { A: 1, NS: 2, CNAME: 5, MX: 15, TXT: 16, AAAA: 28 };
+
+/** NXDOMAIN. The one DNS response code this file treats as load-bearing: it is
+ *  the difference between "that name is not there" and "I could not tell you",
+ *  and a dangling-CNAME finding that confused the two would be a false positive
+ *  aimed at somebody's live site. */
+export const NXDOMAIN = 3;
+
+/**
+ * One DNS question, answered or honestly not.
+ *
+ * `ok: false` means the lookup did not happen — the resolver was unreachable,
+ * slow, or answered something that was not JSON. It is deliberately distinct
+ * from an empty `records`, which means the resolver answered and there is
+ * nothing there. Callers must not read the second as the first.
+ *
+ * @returns {Promise<{ok: boolean, status: number|null, records: string[], cname: string[]}>}
+ */
+export async function resolve(name, type, opts = {}) {
+  const { resolver = RESOLVER, timeout = 5000, fetchImpl = fetch } = opts;
+  const empty = { ok: false, status: null, records: [], cname: [] };
+  if (!TYPE[type]) return empty;
+
+  const controller = new AbortController();
+  const timer = setTimeout(() => controller.abort(), timeout);
+  try {
+    const res = await fetchImpl(`${resolver}?name=${encodeURIComponent(name)}&type=${type}`, {
+      headers: { accept: 'application/dns-json' },
+      signal: controller.signal,
+    });
+    if (!res.ok) return empty;
+    const body = await res.json();
+    const answers = Array.isArray(body?.Answer) ? body.Answer : [];
+    // A trailing dot is how DNS writes a fully qualified name and is noise
+    // everywhere else in this codebase, so it comes off here rather than at
+    // each of the six places that would otherwise have to remember.
+    const data = (rrType) =>
+      answers.filter((a) => a.type === rrType).map((a) => String(a.data ?? '').replace(/\.$/, ''));
+    return {
+      ok: true,
+      status: typeof body?.Status === 'number' ? body.Status : null,
+      records: data(TYPE[type]),
+      // Asking for A on a name that is an alias answers with the CNAME *and*
+      // the addresses it resolves through to. Both are worth having and one
+      // question is cheaper than two, so the alias is always carried back.
+      cname: data(TYPE.CNAME),
+    };
+  } catch {
+    return empty;
+  } finally {
+    clearTimeout(timer);
+  }
+}
+
+/**
+ * Every hostname certificate transparency has ever seen for a domain, and which
+ * log said so.
+ *
+ * `null` means no source answered, which is not the same as a domain with no
+ * certificates and must never be reported as one. That is not a rare branch:
+ * crt.sh alone failed three times in five in testing, which is the reason there
+ * is more than one source here.
+ *
+ * The source is carried back because the report says which log answered. Two
+ * runs of one domain can legitimately list different hosts if different logs
+ * answered them, and a reader comparing those two runs is owed the reason.
+ *
+ * @returns {Promise<{source: string, names: string[]}|null>}
+ */
+export async function certificateNames(domain, opts = {}) {
+  const { sources = CERTIFICATE_SOURCES, timeout = 25000, fetchImpl = fetch } = opts;
+
+  for (const source of sources) {
+    const raw = await askLog(source, domain, { timeout, fetchImpl });
+    if (!raw) continue;
+
+    const names = new Set();
+    for (const entry of raw) {
+      const name = String(entry ?? '').trim().toLowerCase().replace(/\.$/, '');
+      // A wildcard is not a host. `*.example.com` says a certificate covers any
+      // name at that level, not that any particular one exists, and resolving
+      // it asks a question with no answer.
+      if (!name || name.startsWith('*')) continue;
+      if (name === domain || name.endsWith(`.${domain}`)) names.add(name);
+    }
+    return { source: source.name, names: [...names] };
+  }
+  return null;
+}
+
+/** Hostnames that are a fleet rather than a service, collapsed.
+ *
+ *  Certificate transparency is a log of every certificate ever issued, which
+ *  makes it the best free source of hostnames and a terrible list of live ones.
+ *  cloudflare.com's log holds 3,425 distinct names and roughly three thousand
+ *  of them are `ssl2081.cloudflare.com` and its numbered siblings, none of
+ *  which have existed for years.
+ *
+ *  So names are grouped by *shape* — every run of digits becomes `#` — and at
+ *  most `perShape` survive each one. That empties the fleet without a
+ *  hand-written blocklist of somebody's naming convention, and it keeps
+ *  `staging.` and `staging2.` because those are two shapes and both are worth
+ *  a look, while `ssl2081.` and three thousand others become two. */
+export function collapseFleets(names, { perShape = 2 } = {}) {
+  const seen = new Map();
+  const out = [];
+  for (const name of names) {
+    const shape = name.replace(/\d+/g, '#');
+    const count = seen.get(shape) ?? 0;
+    if (count >= perShape) continue;
+    seen.set(shape, count + 1);
+    out.push(name);
+  }
+  return out;
+}
+
+/** The leftmost labels that mean "this was never meant to be public".
+ *
+ *  Kept deliberately tight. `beta.` and `demo.` are missing on purpose: plenty
+ *  of companies run both as products, and a check that reports somebody's
+ *  shipped beta as a leak is the kind of false positive that gets a whole
+ *  report closed. Everything here names an environment, not a product. */
+export const STAGING_LABEL =
+  /^(staging|stage|staging-\w+|dev|develop|development|test|testing|tst|uat|qa|preview|sandbox|wip|scratch)\d*$/;
+
+/** Whether a hostname's leftmost label names a non-production environment. */
+export const looksLikeStaging = (host) => STAGING_LABEL.test(host.split('.')[0]);
+
+/** Candidates worth spending a lookup on, best first.
+ *
+ *  There is a cap, so the order is the difference between finding a leaked
+ *  staging host and finding two hundred names beginning with `a`. Environment
+ *  names first because they are what the checks are for; then the short names,
+ *  because a service a company runs on purpose is usually called something
+ *  short, and the long ones are usually machines. */
+export function rankHosts(names, apex) {
+  const depth = (n) => n.split('.').length;
+  const score = (n) => (looksLikeStaging(n) ? 0 : 1);
+  return [...names]
+    .filter((n) => n !== apex && n !== `www.${apex}`)
+    .sort((a, b) => score(a) - score(b) || depth(a) - depth(b) || a.length - b.length || a.localeCompare(b));
+}

package/src/http.mjs

@@ -81,7 +81,7 @@ export class Fetcher {
    * @returns {Promise<{url: string, status: number, ok: boolean, headers: Headers,
    *                    body: string, location: string|null, ms: number, error?: string}>}
    */
-  async get(url, { method = 'GET', retries = 2 } = {}) {
+  async get(url, { method = 'GET', retries = 2, keepBody = true } = {}) {
     const key = `${method} ${url}`;
     if (this.cache.has(key)) return this.cache.get(key);
 
@@ -171,7 +171,24 @@ export class Fetcher {
       return last;
     });
 
-    this.cache.set(key, promise);
+    // The caller is handed the whole response; the cache may keep less.
+    //
+    // The cache is here so a URL is never fetched twice across checks, and for
+    // most of what a run fetches that means remembering a status and a
+    // content-type — the link sweep, the image sweep and the social-image
+    // sweeps all judge on those alone. Keeping every body as well made the
+    // cache the largest live object in the process: 403 requests on one 25-page
+    // site held 65MB, of which 18MB was still reachable once the run let go.
+    // That is under Node's default heap and over Raycast's, whose commands get
+    // 100MB and were dying on sites this tool audits comfortably from a
+    // terminal.
+    //
+    // So a sweep asks for `keepBody: false` and the body is dropped on the way
+    // into the cache, never on the way out — whoever fetched it reads it in
+    // full, and only a *second* reader of the same URL sees an empty body. The
+    // page crawl, robots.txt, the home page and the host checks all keep
+    // theirs, because something does read those twice.
+    this.cache.set(key, keepBody ? promise : promise.then((res) => ({ ...res, body: '' })));
     return promise;
   }
 
@@ -200,16 +217,16 @@ export class Fetcher {
   }
 
   /** Follow a chain by hand so the number of hops can be reported. */
-  async chain(url, max = 5) {
+  async chain(url, max = 5, { keepBody = true } = {}) {
     const hops = [];
     let current = url;
     for (let i = 0; i < max; i++) {
-      const res = await this.get(current);
+      const res = await this.get(current, { keepBody });
       hops.push({ url: current, status: res.status });
       if (res.status < 300 || res.status >= 400 || !res.location) return { hops, final: res };
       current = new URL(res.location, current).toString();
     }
-    return { hops, final: await this.get(current) };
+    return { hops, final: await this.get(current, { keepBody }) };
   }
 }
 

package/src/options.mjs

@@ -65,6 +65,10 @@ export const OPTIONS = [
   { flag: '--psi-strategy', query: 'psi-strategy', app: true, via: 'Settings → Performance',
     field: { type: 'select', label: 'Measured as', needs: 'ALLOW_PSI',
              choices: [['', 'A phone — what Google indexes with'], ['desktop', 'A desktop']] } },
+  { flag: '--hosts', query: 'hosts', app: true, via: 'Settings → Crawl',
+    field: { type: 'checkbox', label: 'Audit the rest of the domain too', value: '1',
+             needs: 'ALLOW_HOSTS', checked: true,
+             help: 'Asks certificate transparency what other hosts exist, then resolves and fetches them. Adds a few seconds, and more on a domain with hundreds of hosts.' } },
   { flag: '--since', query: null, app: 'not yet — it needs a date picker and a sense of when the last run was, which the window has in the library and does not offer yet' },
   { flag: '--exclude', query: null, app: 'not yet — a list of patterns needs somewhere to live in Settings, and one text field would be worse than nothing' },
 

package/src/report.mjs

@@ -3,6 +3,7 @@
 
 import { byCause, causeScope, sectionOf } from './causes.mjs';
 import { CATEGORIES, categoryOf } from './areas.mjs';
+import { plural } from './text.mjs';
 
 const COLOR = process.env.NO_COLOR === undefined && process.stdout.isTTY;
 const c = (code, s) => (COLOR ? `\x1b[${code}m${s}\x1b[0m` : s);
@@ -122,6 +123,9 @@ export function terminal(findings, meta, { score } = {}) {
   if (!findings.length) {
     lines.push(`  ${c('32', '✓')} nothing to report`);
     lines.push('');
+    // The inventory is not a finding and does not belong to the list above, so
+    // a clean site that was asked about its domain still gets the answer.
+    lines.push(...hostBlock(findings, meta, rule));
     if (score?.passed?.length) lines.push(...passingBlock(score, rule));
     return lines.join('\n');
   }
@@ -189,6 +193,7 @@ export function terminal(findings, meta, { score } = {}) {
     lines.push(...listing(notes));
   }
 
+  lines.push(...hostBlock(findings, meta, rule));
   lines.push(...passingBlock(score, rule));
 
   lines.push(
@@ -198,6 +203,88 @@ export function terminal(findings, meta, { score } = {}) {
   return lines.join('\n');
 }
 
+/** The hosts a finding is actually about, so the inventory can mark them.
+ *
+ *  Read back out of the findings rather than recomputed, because a second
+ *  arithmetic over the same facts is a second chance to disagree with the
+ *  report printed six lines above it. */
+export function flaggedHosts(findings) {
+  const out = new Map();
+  for (const finding of findings) {
+    if (!/^(subdomain-takeover|staging-indexable|duplicate-host)$/.test(finding.id)) continue;
+    try {
+      const host = new URL(finding.url).hostname;
+      // An error outranks a warning on the same host, so a row that is both a
+      // takeover and a duplicate is coloured by the worse of the two.
+      if (finding.level === 'error' || !out.has(host)) out.set(host, finding.level);
+    } catch { /* a finding without a parseable URL marks nothing */ }
+  }
+  return out;
+}
+
+/** What a host row says about itself, in one column.
+ *
+ *  A dangling alias is named rather than reduced to a status, because the name
+ *  it points at is the thing somebody has to go and delete. */
+export function hostSummary(row) {
+  if (row.dangling) return `CNAME → ${row.cname} (gone)`;
+  if (!row.addresses.length) return row.cname ? `CNAME → ${row.cname}` : 'does not resolve';
+  if (!row.checked) return 'not fetched';
+  if (row.redirectsHome) return `${row.status} → the canonical host`;
+  // Where a request for this host's root actually ended up, when it was not
+  // this host's root — another sibling, or a login page. It is the reason the
+  // staging check stayed quiet, so it is said rather than left to be guessed.
+  if (row.landed && row.landed !== row.host) return `${row.status} → ${row.landed}`;
+  if (row.landedPath && row.landedPath !== '/') return `${row.status} → ${row.landedPath}`;
+  if (row.noindex) return `${row.status}, noindex`;
+  if (row.status) return `${row.status}${row.title ? `  ${row.title}` : ''}`;
+  return 'no answer';
+}
+
+/** The rest of the domain, as a table.
+ *
+ *  Printed whether or not anything was wrong with it. "What else is on this
+ *  domain" is worth an answer on a healthy domain too, and it is the answer
+ *  that lets somebody check the findings above rather than take them. */
+function hostBlock(findings, meta, rule) {
+  const inventory = meta?.hosts;
+  if (!inventory?.rows?.length) return [];
+  const flagged = flaggedHosts(findings);
+  const out = [];
+
+  out.push(rule(`Hosts on ${inventory.apex}`));
+  out.push(
+    dim(
+      `  ${plural(inventory.found, 'hostname')} in certificate transparency` +
+        (inventory.source ? ` (${inventory.source})` : '') +
+        `, ${inventory.resolved} of them resolving` +
+        (inventory.capped ? `, ${inventory.capped} not looked up` : ''),
+    ),
+  );
+  out.push('');
+
+  const width = Math.min(38, Math.max(...inventory.rows.map((r) => r.host.length)));
+  for (const row of inventory.rows) {
+    const mark = flagged.has(row.host) ? red('✗') : dim('·');
+    const address = row.addresses[0] ?? '';
+    out.push(
+      `  ${mark} ${row.host.padEnd(width)}  ${dim(address.padEnd(15))} ${hostSummary(row)}`,
+    );
+  }
+  out.push('');
+
+  const facts = [
+    ['Nameservers', inventory.nameservers],
+    ['Mail', inventory.mail],
+    ['Policies', inventory.policies],
+  ];
+  for (const [label, values] of facts) {
+    if (values?.length) out.push(dim(`  ${label.padEnd(12)} ${values.slice(0, 4).join(', ')}`));
+  }
+  out.push('');
+  return out;
+}
+
 /** What passed, and what never came up.
  *
  *  A report that only ever lists faults reads as a list of everything the tool
@@ -320,6 +407,7 @@ export function markdown(findings, meta, { score } = {}) {
     out.push(...listing(notes));
   }
 
+  out.push(...hostMarkdown(findings, meta));
   out.push(...passingMarkdown(score));
 
   out.push('---');
@@ -333,6 +421,48 @@ export function markdown(findings, meta, { score } = {}) {
   return out.join('\n');
 }
 
+/** The rest of the domain, in a file somebody sends to a client.
+ *
+ *  A table rather than prose, because the useful thing to do with it is read
+ *  down the first column and recognise a name nobody meant to leave running. */
+function hostMarkdown(findings, meta) {
+  const inventory = meta?.hosts;
+  if (!inventory?.rows?.length) return [];
+  const flagged = flaggedHosts(findings);
+  const out = [];
+
+  out.push(`## Hosts on ${inventory.apex}`);
+  out.push('');
+  out.push(
+    `${plural(inventory.found, 'hostname')} in certificate transparency` +
+      (inventory.source ? ` (${inventory.source})` : '') +
+      `, ${inventory.resolved} of them resolving` +
+      (inventory.capped ? `, ${inventory.capped} not looked up` : '') +
+      '. Discovered from the log and then verified — every row below was resolved, and every row ' +
+      'with a status was fetched.',
+  );
+  out.push('');
+  out.push('| Host | Address | What it serves |');
+  out.push('|---|---|---|');
+  for (const row of inventory.rows) {
+    const name = flagged.has(row.host) ? `**${row.host}**` : row.host;
+    out.push(`| ${name} | ${row.addresses[0] ?? '—'} | ${hostSummary(row)} |`);
+  }
+  out.push('');
+
+  const facts = [
+    ['Nameservers', inventory.nameservers],
+    ['Mail', inventory.mail],
+    ['Policies', inventory.policies],
+  ];
+  const shown = facts.filter(([, values]) => values?.length);
+  if (shown.length) {
+    for (const [label, values] of shown) out.push(`**${label}** — ${values.slice(0, 4).join(', ')}`);
+    out.push('');
+  }
+  return out;
+}
+
 /** The score, in a file somebody sends to a client. */
 function scoreMarkdown(score) {
   if (score?.score === null || score?.score === undefined) {
@@ -731,6 +861,20 @@ export function csv(findings, meta, { score } = {}) {
     rows.push(['not-checked', check.id, check.pass, '', '', 'yes', '', '', '', '', check.why, '']);
   }
 
+  // The rest of the domain, when the run was asked to look. A new level rather
+  // than a second table with its own header: a CSV holding two shapes is a CSV
+  // that no spreadsheet and no script can read in one go, and this file already
+  // answers "what else did the run see" the same way for passed and skipped
+  // checks. Filtering on error/warn/info is still untouched.
+  for (const row of meta?.hosts?.rows ?? []) {
+    rows.push([
+      'host', 'host', row.host, `https://${row.host}/`, '',
+      row.noindex ? 'no' : 'yes', '', '', '', '',
+      [row.addresses.join(' '), hostSummary(row)].filter(Boolean).join(' · '),
+      '',
+    ]);
+  }
+
   return `\uFEFF${[columns, ...rows].map((row) => row.map(cell).join(',')).join('\r\n')}\r\n`;
 }
 
@@ -836,6 +980,53 @@ export function reportParts(findings, meta, { backHref, backLabel = 'New audit',
   }`;
   };
 
+  // --- The rest of the domain ---------------------------------------------
+  // A list rather than the findings table, because these rows are not findings:
+  // most of them are a healthy domain going about its business, and rendering
+  // them in the table that means "something is wrong" would make a mail server
+  // look like a problem. The two or three that *are* findings borrow the pill
+  // and the colour from the section above, so the eye connects them.
+  const hostTable = () => {
+    const inventory = meta?.hosts;
+    if (!inventory?.rows?.length) return '';
+    const flagged = flaggedHosts(findings);
+    const facts = [
+      ['Nameservers', inventory.nameservers],
+      ['Mail', inventory.mail],
+      ['SPF and DMARC', inventory.policies],
+    ].filter(([, values]) => values?.length);
+
+    const row = (host) => {
+      const level = flagged.get(host.host);
+      const address = host.addresses[0] ?? '';
+      return `<li class="${level ?? 'quiet'}">
+      <span class="pill ${level ?? 'none'}">${level ? LABEL[level] : '·'}</span>
+      <b>${esc(host.host)}</b>
+      <span class="addr">${esc(address)}</span>
+      <span class="detail">${esc(hostSummary(host))}</span>
+    </li>`;
+    };
+
+    return `
+  <h2 id="hosts"><span>Hosts on ${esc(inventory.apex)}</span><span class="rule"></span><span class="tick">${inventory.rows.length}</span></h2>
+  <p class="lede">${plural(inventory.found, 'hostname')} in certificate transparency${
+    inventory.source ? ` (${esc(inventory.source)})` : ''
+  }, ${inventory.resolved} of them resolving${
+    inventory.capped ? `, ${inventory.capped} not looked up` : ''
+  }. The log is only where the names came from — every row below was resolved, and every row with a status was fetched.</p>
+  <ul class="hosts">${inventory.rows.map(row).join('')}</ul>
+  ${
+    facts.length
+      ? `<ul class="dnsfacts">${facts
+          .map(
+            ([label, values]) =>
+              `<li><b>${esc(label)}</b><span>${values.slice(0, 4).map((v) => `<code>${esc(v)}</code>`).join(' ')}</span></li>`,
+          )
+          .join('')}</ul>`
+      : ''
+  }`;
+  };
+
   const section = ({ name, entries: list }) => {
     if (!list.length) return '';
     const slug = name.toLowerCase().replace(/[^a-z]+/g, '-');
@@ -1046,6 +1237,47 @@ export function reportParts(findings, meta, { backHref, backLabel = 'New audit',
   .skipped b { font-weight: 500; font-size: .88rem; }
   .skipped code { font: 400 11.5px/1.7 ui-monospace, SFMono-Regular, "SF Mono", Menlo, Consolas, monospace; color: var(--faint); margin-right: .45rem; }
 
+  /* --- Hosts ---------------------------------------------------------- */
+  /* One row per host, on a grid so the names, the addresses and what each one
+     serves line up into three columns somebody can read down. It collapses to
+     a stack on a phone rather than scrolling sideways: a hostname is the thing
+     being looked for and it must never be the column that goes off the edge. */
+  .hosts { list-style: none; margin: 0 0 1.6rem; padding: 0; display: grid; gap: .4rem; }
+  .hosts li {
+    display: grid; align-items: baseline; gap: .3rem .75rem;
+    grid-template-columns: 5.25rem minmax(0, 15rem) minmax(0, 9.5rem) minmax(0, 1fr);
+    padding: .6rem .85rem; border: 1px solid var(--line); border-radius: 8px;
+    background: var(--panel);
+  }
+  .hosts li.error { border-color: var(--error); background: var(--error-bg); }
+  .hosts li.warn { border-color: var(--warn); background: var(--warn-bg); }
+  .hosts b { font-weight: 550; font-size: .9rem; word-break: break-all; }
+  .hosts .addr, .hosts .detail {
+    font: 400 12px/1.6 ui-monospace, SFMono-Regular, "SF Mono", Menlo, Consolas, monospace;
+    color: var(--muted);
+  }
+  .hosts .addr { color: var(--faint); font-variant-numeric: tabular-nums; }
+  .hosts .pill.none {
+    background: transparent; border-color: transparent;
+    color: var(--faint); padding-left: 0; padding-right: 0;
+  }
+  .dnsfacts { list-style: none; margin: 0 0 2.5rem; padding: 0; display: grid; gap: .45rem; }
+  .dnsfacts li { display: grid; grid-template-columns: minmax(7.5rem, auto) 1fr; gap: .3rem .75rem; align-items: baseline; }
+  .dnsfacts b {
+    font-size: .715rem; font-weight: 600; text-transform: uppercase;
+    letter-spacing: .075em; color: var(--muted);
+  }
+  .dnsfacts code {
+    font: 400 11.5px/1.7 ui-monospace, SFMono-Regular, "SF Mono", Menlo, Consolas, monospace;
+    color: var(--faint); word-break: break-all; margin-right: .5rem;
+  }
+  @media (max-width: 640px) {
+    .hosts li { grid-template-columns: 5.25rem minmax(0, 1fr); }
+    .hosts .addr { grid-column: 2; }
+    .hosts .detail { grid-column: 1 / -1; }
+    .dnsfacts li { grid-template-columns: 1fr; }
+  }
+
   /* --- Tally ---------------------------------------------------------- */
   .tally { display: grid; grid-template-columns: repeat(3, 1fr); gap: .7rem; margin: 0 0 3rem; }
   .tally div {
@@ -1248,6 +1480,8 @@ export function reportParts(findings, meta, { backHref, backLabel = 'New audit',
       : `<div class="clean"><b>Nothing to report</b><span>Every check passed on all ${meta.pages ?? 0} pages.</span></div>`
   }
 
+  ${hostTable()}
+
   ${passing()}
 
   <footer>

package/src/score.mjs

@@ -193,6 +193,12 @@ const CHECKLIST = {
   'tls-expired': { worst: 'error', scope: 'site', needs: 'tls', pass: 'The TLS certificate is valid' },
   'tls-expiring': { worst: 'warn', scope: 'site', needs: 'tls', pass: 'The TLS certificate is not about to expire' },
   'header-strict-transport-security': { worst: 'warn', scope: 'site', needs: 'https', pass: 'HSTS is set' },
+  // The domain rather than the site. All three need --hosts, and all three are
+  // one yes or no for the whole domain rather than a share of the crawl: a
+  // leaked staging copy is not more wrong because the site has more pages.
+  'subdomain-takeover': { worst: 'error', scope: 'site', needs: 'hosts', pass: 'No subdomain points at a service that is gone' },
+  'staging-indexable': { worst: 'warn', scope: 'site', needs: 'hosts', pass: 'No staging copy is open to the index' },
+  'duplicate-host': { worst: 'warn', scope: 'site', needs: 'hosts', pass: 'No other host serves the same site again' },
   'url-uppercase': { worst: 'warn', scope: 'page', pass: 'No URL has uppercase in its path' },
   'url-space': { worst: 'warn', scope: 'page', pass: 'No URL has a space in it' },
   uncompressed: { worst: 'warn', scope: 'page', pass: 'HTML arrives compressed' },
@@ -343,6 +349,8 @@ export function scoreRun(findings, { pages = 0, applicable = {} } = {}) {
       area: categoryOf(r.id),
       pass: r.check.pass,
       why: WHY_SKIPPED[r.check.needs] ?? 'Not applicable to this run.',
+      // Absent when nothing can be pressed, which is most of them.
+      ...(ENABLED_BY[r.check.needs] ? { enabledBy: ENABLED_BY[r.check.needs] } : {}),
     }));
 
   return {
@@ -385,6 +393,31 @@ const WHY_SKIPPED = {
   ogImage: 'No page declares an og:image.',
   twitterImage: 'No page declares a twitter:image of its own.',
   compareAs: 'The pages were not fetched a second time — run with --compare-as.',
+  hosts: 'The rest of the domain was not enumerated — run with --hosts.',
+};
+
+/** The option that would have let a skipped check run, where one exists.
+ *
+ *  A skip has two very different causes and only one of them is anybody's to
+ *  fix. "No page declares hreflang" is a fact about the site — there is nothing
+ *  to press. "Outbound links were not checked" is a run that was not asked to,
+ *  and asking is one flag away.
+ *
+ *  Saying which is which here rather than in each front end, because the engine
+ *  is what decided the check was skipped and a client re-deriving that from the
+ *  prose in WHY_SKIPPED would be parsing an English sentence for a flag name.
+ *  A front end that can offer the re-run offers it; one that cannot ignores
+ *  this, and nothing about the report changes.
+ *
+ *  Only the reasons a *run* controls appear. `redirects` and `compareAs` are
+ *  deliberately absent even though both are flags: one needs a file that only
+ *  the person who did the migration has, and the other needs a second identity
+ *  to fetch as. Offering a button that cannot be pressed without an argument
+ *  is worse than offering nothing. */
+export const ENABLED_BY = {
+  external: '--check-external',
+  hosts: '--hosts',
+  psi: '--psi',
 };
 
 /** The same sum, once per area, so "where is this site weak" is answerable

package/src/serve.mjs

@@ -44,6 +44,11 @@ export async function serve({ port = 4321, host = '127.0.0.1', maxPages, allowed
     // person whose account it is; a deployed Worker leaves this unset, where
     // `?search-console=` would hand a stranger somebody else's traffic data.
     ALLOW_SEARCH_CONSOLE: '1',
+    // Same reasoning again: enumerating the domain calls crt.sh, which is free,
+    // unauthenticated and rate-limited by IP. On the loopback address that
+    // budget is the caller's own to spend. A deployed Worker leaves this unset,
+    // where every visitor would be spending one shared address's allowance.
+    ALLOW_HOSTS: '1',
     // This is Node, so `node:tls` works and the certificate checks are real.
     // Cloudflare leaves this unset and gets the `tls-not-checked` note instead.
     // The macOS window talks to this server, and was being handed a hosted

package/src/site.mjs

@@ -6,6 +6,10 @@ import { parseRobots, robotsVerdict } from './robots.mjs';
 import { aiAccess, describeAccess } from './agents-ai.mjs';
 import { parseHtml } from './parse.mjs';
 import { schemaNodes, seriesOf, paginatedCanonical } from './checks.mjs';
+import { similarity } from './dupes.mjs';
+import {
+  resolve as resolveDns, certificateNames, collapseFleets, rankHosts, looksLikeStaging, NXDOMAIN,
+} from './dns.mjs';
 import { plural } from './text.mjs';
 
 // Two weeks is enough to renew by hand if the automation has quietly stopped,
@@ -375,7 +379,7 @@ export async function siteChecks(origin, fetcher, pages, opts = {}) {
   const targets = all.slice(0, limit);
   opts.onProgress?.({ phase: 'links', detail: `${targets.length} distinct targets to check` });
   const results = await mapLimit(targets, 6, async (target) => {
-    const res = await fetcher.get(target);
+    const res = await fetcher.get(target, { keepBody: false });
     opts.onProgress?.({ phase: 'links', status: res.status, ms: res.ms, url: target });
     const type = res.headers.get('content-type') ?? '';
     // A third question the same response answers — and the only place it can be
@@ -495,9 +499,9 @@ export async function siteChecks(origin, fetcher, pages, opts = {}) {
   const imageTargets = [...imageSources.values()].slice(0, imageLimit).map((entry) => entry.src);
   opts.onProgress?.({ phase: 'images', detail: `${imageTargets.length} distinct images to check` });
   const imageResults = await mapLimit(imageTargets, 6, async (src) => {
-    let res = await fetcher.get(src, { method: 'HEAD' });
+    let res = await fetcher.get(src, { method: 'HEAD', keepBody: false });
     // Some hosts answer HEAD with 405 or 501 and serve the file perfectly well.
-    if (res.status === 405 || res.status === 501) res = await fetcher.get(src);
+    if (res.status === 405 || res.status === 501) res = await fetcher.get(src, { keepBody: false });
     opts.onProgress?.({ phase: 'images', status: res.status, ms: res.ms, url: src });
     return { src, status: res.status, error: res.error };
   });
@@ -586,8 +590,8 @@ export async function siteChecks(origin, fetcher, pages, opts = {}) {
   }
   const schemaTargets = [...schemaImages.keys()].slice(0, opts.maxImageChecks ?? 200);
   const schemaResults = await mapLimit(schemaTargets, 4, async (href) => {
-    let res = await fetcher.get(href, { method: 'HEAD' });
-    if (res.status === 405 || res.status === 501) res = await fetcher.get(href);
+    let res = await fetcher.get(href, { method: 'HEAD', keepBody: false });
+    if (res.status === 405 || res.status === 501) res = await fetcher.get(href, { keepBody: false });
     return { href, status: res.status, error: res.error };
   });
   const deadSchemaImages = schemaResults.filter(
@@ -729,7 +733,7 @@ export async function siteChecks(origin, fetcher, pages, opts = {}) {
   // Conservative about what counts as broken, for the same reason as the image
   // sweep: 403 is hotlink protection working, not a missing file.
   const ogResults = await mapLimit([...ogImages.keys()], 4, async (src) => {
-    const { final } = await fetcher.chain(src);
+    const { final } = await fetcher.chain(src, 5, { keepBody: false });
     return { src, final };
   });
   for (const { src, final } of ogResults) {
@@ -764,7 +768,7 @@ export async function siteChecks(origin, fetcher, pages, opts = {}) {
     twitterImages.set(src, page.url);
   }
   const twitterResults = await mapLimit([...twitterImages.keys()], 4, async (src) => {
-    const { final } = await fetcher.chain(src);
+    const { final } = await fetcher.chain(src, 5, { keepBody: false });
     return { src, final };
   });
   for (const { src, final } of twitterResults) {
@@ -777,3 +781,292 @@ export async function siteChecks(origin, fetcher, pages, opts = {}) {
 
   return out;
 }
+
+// --- Every other host on the domain -----------------------------------------
+//
+// The rest of this file audits the site it was pointed at. This asks a
+// different question: what *else* is on this domain, and is any of it damaging
+// the site that was.
+//
+// Three things come out of it, and all three are invisible to a crawl of the
+// site itself, which is the reason it exists at all:
+//
+//   * a staging copy nobody remembered to close, indexable, competing with
+//     production for its own results and publishing whatever was being tested;
+//   * a subdomain whose CNAME points at a service that is gone, which anybody
+//     can claim and then serve from the client's own domain;
+//   * a second host serving the same site again, splitting its signals.
+//
+// The discovery is certificate transparency and the verification is this tool's
+// own fetcher, and that split is the whole design. CT is a log of every
+// certificate ever issued: it is the only free, keyless, complete source of
+// hostnames, and it is a terrible list of *live* ones — three quarters of what
+// it returns for a large domain stopped existing years ago. So nothing from the
+// log is ever reported. It produces candidates; DNS and an HTTP request decide
+// which of them are facts. A finding here has been resolved and fetched.
+//
+// Off by default. It is one slow third-party lookup plus a lookup per candidate,
+// and a crawl should not quietly spend that.
+
+/** Names that are the site itself rather than a sibling. */
+const canonicalNames = (base) => new Set([base.hostname, base.hostname.replace(/^www\./, ''),
+  `www.${base.hostname.replace(/^www\./, '')}`]);
+
+/** Whether a response forbids indexing, by either of the two routes Google
+ *  reads it from. The same test the soft-404 check makes, for the same reason:
+ *  a noindexed copy is a mistake somebody already mitigated. */
+function noindexed(res) {
+  const meta = res.body?.match(/<meta[^>]+name=["']robots["'][^>]*>/i)?.[0] ?? '';
+  return /noindex/i.test(meta) || /noindex/i.test(res.headers?.get?.('x-robots-tag') ?? '');
+}
+
+/**
+ * Sibling hosts, and what is wrong with them.
+ *
+ * Returns findings *and* the inventory behind them, because a reader who is
+ * told a domain has a leaked staging host is owed the list the tool actually
+ * looked at — including the part it did not reach.
+ *
+ * @returns {Promise<{findings: object[], hosts: object|null}>}
+ */
+export async function hostChecks(origin, fetcher, opts = {}) {
+  const out = [];
+  const base = new URL(origin);
+  const apex = base.hostname.replace(/^www\./, '');
+
+  // An IP address and a bare hostname have no domain to enumerate, and asking
+  // the log about one is a question with no sensible answer. The same guard the
+  // www variants use, for the same reason.
+  const isAddress = /^\[?[\d.:]+\]?$/.test(base.hostname);
+  if (isAddress || !apex.includes('.')) return { findings: out, hosts: null };
+
+  // Injectable so the tests never touch the network, and so a network that
+  // blocks either service can be pointed somewhere that answers.
+  const lookup = opts.resolveDns ?? resolveDns;
+  const fromLog = opts.certificateNames ?? certificateNames;
+  const dnsOpts = opts.dnsOptions ?? {};
+
+  const log = await fromLog(apex, dnsOpts);
+  if (log === null) {
+    // A log that did not answer is not a domain with no other hosts. Saying so
+    // is the same rule `tls-not-checked` and `sitemap-not-checked` follow: a
+    // check that could not run says it could not run, because a missing finding
+    // reads exactly like a passing one.
+    out.push(f('info', 'hosts-not-checked', 'Other hosts on this domain were not enumerated',
+      'No certificate transparency log answered, so this run cannot say whether the domain serves ' +
+        'anything besides the site that was audited. Nothing about the site itself is affected. Try ' +
+        'again — these logs are free and unauthenticated, and they rate-limit and time out ' +
+        'accordingly.', origin));
+    return { findings: out, hosts: null };
+  }
+
+  const mine = canonicalNames(base);
+  const ranked = rankHosts(collapseFleets(log.names), apex);
+  const limit = opts.maxHostChecks ?? 40;
+  const targets = ranked.slice(0, limit);
+  opts.onProgress?.({ phase: 'hosts', detail: `${targets.length} of ${ranked.length} hostnames to resolve` });
+
+  if (ranked.length > targets.length) {
+    out.push(f('info', 'host-sweep-capped', `${ranked.length - targets.length} hostnames were not looked up`,
+      `The log named ${ranked.length} distinct hosts for this domain and the sweep stops at ${limit}, ` +
+        'environment names first. Raise it with maxHostChecks in the config — the rest of this section ' +
+        'describes only what was actually resolved.', origin));
+  }
+
+  // --- Resolve ------------------------------------------------------------
+  const resolved = await mapLimit(targets, 8, async (host) => {
+    const a = await lookup(host, 'A', dnsOpts);
+    const alias = a.cname.at(-1) ?? null;
+
+    // A dangling CNAME: the alias is there and what it points at is not.
+    //
+    // Most resolvers answer NXDOMAIN for the whole query when the chain ends in
+    // a name that does not exist, so usually one lookup settles it. Some answer
+    // NOERROR with no address instead, so that case is chased explicitly rather
+    // than guessed at — this is the finding that accuses somebody's live domain
+    // of being claimable, and it is not allowed to be approximate.
+    let dangling = false;
+    if (alias) {
+      if (a.ok && a.status === NXDOMAIN) dangling = true;
+      else if (a.ok && !a.records.length) {
+        const chased = await lookup(alias, 'A', dnsOpts);
+        dangling = chased.ok && chased.status === NXDOMAIN;
+      }
+    }
+    opts.onProgress?.({ phase: 'hosts', url: host, detail: a.records.length ? a.records[0] : 'no address' });
+    return { host, addresses: a.records, cname: alias, dangling, live: a.ok && a.records.length > 0 };
+  });
+
+  for (const row of resolved) {
+    if (!row.dangling) continue;
+    out.push(f('error', 'subdomain-takeover', `${row.host} points at a service that no longer exists`,
+      `It is a CNAME to ${row.cname}, and that name does not resolve — the provider it was hosted on has ` +
+        'released it. Anybody can register the same name at that provider and serve whatever they like ' +
+        `from ${row.host}, which is a host on this domain and inherits its reputation. Delete the DNS ` +
+        'record, or point it back at something you own.', `https://${row.host}/`));
+  }
+
+  // --- Fetch what resolves ------------------------------------------------
+  // Only live hosts, and only their home page. The point is to establish what a
+  // search engine would find at each one, which the first page answers.
+  const live = resolved.filter((r) => r.live && !mine.has(r.host));
+  const fetchLimit = opts.maxHostFetches ?? 25;
+  const probed = await mapLimit(live.slice(0, fetchLimit), 4, async (row) => {
+    const { final } = await fetcher.chain(`https://${row.host}/`);
+    const type = final.headers?.get?.('content-type') ?? '';
+    const isHtml = /text\/html/i.test(type);
+    let landed = null;
+    let landedPath = null;
+    try {
+      const at = new URL(final.url);
+      landed = at.hostname;
+      landedPath = at.pathname;
+    } catch { /* an unparseable final URL is not a host to compare */ }
+    const doc = final.ok && isHtml && final.body ? parseHtml(final.body, final.url) : null;
+    return {
+      ...row,
+      status: final.status,
+      error: final.error ?? null,
+      isHtml,
+      // A sibling that redirects to the canonical host is the correct
+      // arrangement, not a finding — it is how a company parks an old name.
+      redirectsHome: Boolean(landed && mine.has(landed)),
+      // Where asking for this host's home page actually ended up.
+      landed,
+      landedPath,
+      // The one condition the staging and duplicate checks actually need: this
+      // host answered for its own root. Anything else — a redirect off to
+      // another host, or a bounce to a login page — means the host is not
+      // serving a copy of anything, whatever the final response says.
+      //
+      // Both real false positives this check has had were this, in different
+      // disguises. `dev.gtm.github.com` bounces to `/login`, which answers 200
+      // with HTML and no noindex. `dev.jquery.com` 301s to `bugs.jquery.com`,
+      // a different sibling, so the old redirectsHome test — which only knew
+      // about the canonical host — let it through and the landing path was `/`.
+      // One condition covers both and needs no vocabulary of login paths.
+      servesOwnRoot: landed === row.host && landedPath === '/',
+      noindex: final.ok ? noindexed(final) : false,
+      title: doc?.title ?? null,
+      canonical: doc?.canonical?.[0] ?? null,
+      fingerprint: doc?.fingerprint ?? null,
+    };
+  });
+
+  // --- A staging copy that is open to the index ---------------------------
+  //
+  // Narrow on purpose, and every clause is doing work. The name has to be an
+  // environment rather than a product — `beta.` and `demo.` are deliberately
+  // not in that list, because companies ship both. It has to actually serve a
+  // page. It must not already be handled: a `noindex`, a canonical pointing at
+  // production, or a robots.txt that disallows crawling are all somebody having
+  // thought about this, and reporting them would be the cry-wolf that gets a
+  // whole report ignored.
+  const staged = new Set();
+  for (const row of probed) {
+    if (!looksLikeStaging(row.host) || row.status !== 200 || !row.isHtml) continue;
+    if (row.noindex || !row.servesOwnRoot) continue;
+    // Its own robots.txt, not the site's. A staging host that blocks crawlers
+    // is a staging host somebody closed.
+    const robots = await fetcher.get(`https://${row.host}/robots.txt`);
+    if (robots.ok && !robotsVerdict(parseRobots(robots.body), '/').allowed) continue;
+    // A canonical pointing at production is the other legitimate arrangement.
+    let canonicalHost = null;
+    try {
+      canonicalHost = row.canonical ? new URL(row.canonical, `https://${row.host}/`).hostname : null;
+    } catch { /* a malformed canonical is not a defence */ }
+    if (canonicalHost && mine.has(canonicalHost)) continue;
+
+    staged.add(row.host);
+    out.push(f('warn', 'staging-indexable', `${row.host} is a live, indexable copy of the site`,
+      `It answers 200 with HTML${row.title ? ` and is titled “${row.title}”` : ''}, carries no noindex, ` +
+        'and nothing in its robots.txt or its canonical keeps it out of the index. A staging host that ' +
+        'Google can reach competes with production for its own results and publishes whatever is being ' +
+        'tested on it. Add a noindex, disallow it in robots.txt, or put it behind authentication.',
+      `https://${row.host}/`));
+  }
+
+  // --- The same site again, on another host -------------------------------
+  //
+  // Compared body to body rather than by title, for the same reason
+  // `duplicate-content` is: two hosts serving one site is a real split of the
+  // signals, and two hosts that happen to share a title is a coincidence.
+  // Silent when the sibling has already been reported as staging — one fault
+  // gets one finding.
+  const home = await fetcher.get(`${origin}/`);
+  const homeDoc = home.ok && /text\/html/i.test(home.headers.get('content-type') ?? '')
+    ? parseHtml(home.body, `${origin}/`)
+    : null;
+  if (homeDoc?.fingerprint) {
+    for (const row of probed) {
+      if (staged.has(row.host) || row.noindex || !row.servesOwnRoot) continue;
+      if (row.status !== 200 || !row.fingerprint) continue;
+      let canonicalHost = null;
+      try {
+        canonicalHost = row.canonical ? new URL(row.canonical, `https://${row.host}/`).hostname : null;
+      } catch { /* as above */ }
+      if (canonicalHost && mine.has(canonicalHost)) continue;
+      if (similarity(homeDoc.fingerprint, row.fingerprint) < 0.9) continue;
+
+      out.push(f('warn', 'duplicate-host', `${row.host} serves the same site again`,
+        `Its home page is the same page as ${origin}/, it is indexable, and its canonical does not point ` +
+          'back. Two hosts serving one site split every signal the site earns between them, and Google ' +
+          'picks which one to show. Redirect it to the canonical host, or make it say so with a canonical.',
+        `https://${row.host}/`));
+    }
+  }
+
+  // --- The inventory ------------------------------------------------------
+  // Shipped whether or not anything was wrong, because "what else is on this
+  // domain" is a question worth an answer on a healthy domain too.
+  const [ns, mx, txt] = await Promise.all([
+    lookup(apex, 'NS', dnsOpts), lookup(apex, 'MX', dnsOpts), lookup(apex, 'TXT', dnsOpts),
+  ]);
+  const byHost = new Map(probed.map((p) => [p.host, p]));
+
+  return {
+    findings: out,
+    hosts: {
+      apex,
+      // Which log answered. Two runs of one domain can list different hosts
+      // because different sources answered them, and a reader comparing those
+      // two runs is owed the reason rather than left to suspect the domain
+      // changed.
+      source: log.source,
+      found: ranked.length,
+      resolved: resolved.filter((r) => r.live).length,
+      capped: Math.max(0, ranked.length - targets.length),
+      nameservers: ns.records,
+      // An MX record's data is a preference and a host — "10 mx.example.net" —
+      // and the number is a mail-routing detail nobody reads an SEO report for.
+      mail: mx.records.map((r) => r.replace(/^\d+\s+/, '')),
+      // Only the TXT records that say something about who the domain talks to.
+      // The rest are verification strings for a dozen SaaS products and are
+      // nobody's business in an SEO report.
+      // TXT records arrive quoted, and a long one arrives as several quoted
+      // chunks that are one string joined — which is the wire format, not
+      // something to print at somebody.
+      policies: txt.records
+        .map((r) => r.replace(/"\s*"/g, '').replace(/^"|"$/g, ''))
+        .filter((r) => /^v=(spf1|DMARC1)/i.test(r)),
+      rows: resolved.map((row) => {
+        const seen = byHost.get(row.host);
+        return {
+          host: row.host,
+          addresses: row.addresses,
+          cname: row.cname,
+          dangling: row.dangling,
+          status: seen?.status ?? null,
+          title: seen?.title ?? null,
+          landedPath: seen?.landedPath ?? null,
+          landed: seen?.landed ?? null,
+          redirectsHome: seen?.redirectsHome ?? false,
+          noindex: seen?.noindex ?? false,
+          // Said explicitly rather than inferred from a null status, which
+          // would read as "did not answer" for a host nobody asked.
+          checked: Boolean(seen),
+        };
+      }),
+    },
+  };
+}