@goflag/cli 0.2.7 → 0.2.8

package/README.md

@@ -16,7 +16,7 @@ on the site. A human cannot check those on 400 pages. That is the whole job.
 ## What it checks
 
 - **Broken links** — scrapes every crawled page, dedupes targets globally (a footer link on 500 pages is probed once), and checks each with `HEAD`→`GET` fallback, redirect-chain/loop detection, soft-404 and anti-bot (403/429) triage. Broken links are mapped back to the pages that reference them.
-- **Missing translation pages** — builds a `route × locale` matrix and flags every route that exists in one locale but is missing in another, plus hreflang reciprocity gaps (`A → B` with no `B → A`), missing `x-default`, and locale tags naming a language, script or region that does not exist. Rows are keyed by pathname, unless your sitemap declares its translation clusters with `xhtml:link` — then those declarations decide which URLs are one page, whatever their slugs, and the run reports `diagnostics.declaredClusters`.
+- **Missing translation pages** — builds a `route × locale` matrix and flags every route that exists in one locale but is missing in another, plus hreflang reciprocity gaps (`A → B` with no `B → A`), missing `x-default`, and locale tags naming a language, script or region that does not exist. Rows are keyed by pathname, unless the site declares its translation clusters — either with `xhtml:link` in the sitemap, or with reciprocal `hreflang` between two crawled pages' `<head>`. Then those declarations decide which URLs are one page, whatever their slugs, and the run reports `diagnostics.declaredClusters`. The sitemap source survives `--coverage structural`, where the two locales of a slug-translating family are rarely both sampled; the `<head>` source needs both pages in hand and is what fixes the common site that declares properly there and nothing in its sitemap.
 - **robots.txt policy** — flags a site that forbids crawling while its pages ask to be indexed. The two declarations cannot both hold, and robots.txt wins, so the meta tag is never even read.
 - **SEO metadata** — lints each page's `<head>` for the handful of mistakes that actually hurt in search and social and are invisible in a browser: missing/oversized `<title>` and description, missing/relative canonical, missing `og:title`/`og:description`/`og:image`, missing viewport, and contradictory `robots`/`googlebot`/`X-Robots-Tag` directives.
 - **Unreachable pages** — a page the crawl reached that answered non-2xx, or that never answered at all (timeout, reset, DNS failure — recorded as `status: 0`), is reported in `unreachablePages`. goflag asks a second time, from the back of the queue, before calling a page unreachable. Each one is an `error`-severity finding: it turns the verdict red, counts toward `--max-debt`, and fails the build even under `--fail-on error`. A page that silently dropped out of the audited set is what poisons a baseline, so it is reported rather than warned about.
@@ -146,6 +146,18 @@ authority nobody assigned. Write your consumer against that. The same document
 ships inside the package as `rules.json`, if you would rather read it than run
 anything.
 
+The flag list below is data too:
+
+```sh
+npx @goflag/cli flags > flags.json
+```
+
+Same idea, one step further: `flags` prints the very table `goflag --help` is
+rendered from and the argument parser dispatches on, so a flag cannot exist in
+one and not the other. Each entry carries its long and short forms, its argument
+placeholder, its default, the group it belongs to and any flag it requires. It
+ships inside the package as `flags.json` as well.
+
 It deliberately omits the message a finding prints — that is built at audit time
 from what the page actually says, so a static copy would be a sample rather than
 a fact, and a sample presented as the truth is how this project's own
@@ -424,6 +436,17 @@ goflag https://example.com --ignore-holes /legal --ignore-holes "/blog/**"
 Suppressed gaps are counted in `diagnostics.ignoredHoles`, so a quiet report
 never means "nothing was wrong" by accident.
 
+The opposite failure is quieter still. A cell in the matrix can be filled by an
+`hreflang` on another page and by nothing else — not crawled, not listed in your
+sitemap — and a filled cell reads as a translation that exists. A page
+advertising a translation the site does not serve therefore plugs the very gap
+it should reveal. Those cells are counted in
+`diagnostics.unverifiedAlternates`, and goflag deliberately does **not** act on
+them: refusing to believe an alternate the sitemap omits would invent holes on
+every site using `@goflag/next`'s `sitemap: false` on purpose. Read that number
+next to `missingTranslations` — a `0` beside a non-zero count is the one place
+this report can be quietly wrong.
+
 ## In CI
 
 Two moments are worth auditing, and they answer different questions.
@@ -455,7 +478,7 @@ Three rules, whichever you use:
 stages: [build, deploy, audit]
 
 variables:
-  GOFLAG_VERSION: "0.2.7"
+  GOFLAG_VERSION: "0.2.8"
 
 # Before the merge: build the branch, let goflag boot it, audit that.
 seo:mr:
@@ -507,7 +530,7 @@ jobs:
       - run: corepack enable && pnpm install --frozen-lockfile
       - run: pnpm build
       - run: |
-          npx --yes @goflag/cli@0.2.7 http://localhost:3000 \
+          npx --yes @goflag/cli@0.2.8 http://localhost:3000 \
             --start "pnpm start" --static --no-external \
             --baseline .goflag/baseline.json --regressions-only --max-debt 41 \
             --report goflag-report.json

package/dist/{chunk-UOKYUKHQ.js → chunk-SKX7UCGC.js}

@@ -1856,13 +1856,18 @@ function buildI18nMatrix(pages, options = {}) {
   const grid = /* @__PURE__ */ new Map();
   const locales = /* @__PURE__ */ new Set();
   const routes = /* @__PURE__ */ new Set();
+  const declaredSet = new Set(options.declaredUrls ?? []);
   for (const locale of options.locales ?? []) {
     const tag = locale.trim().toLowerCase();
     if (tag) locales.add(tag);
   }
-  function rowOf(url, pathnameRoute) {
+  function rowOf2(url, pathnameRoute) {
     return options.clusterRouteOf?.(url) ?? pathnameRoute;
   }
+  function sourceOf(url) {
+    if (inspectedByUrl.has(url)) return "crawled";
+    return declaredSet.has(url) ? "sitemap" : "alternate";
+  }
   function record(route, locale, url) {
     routes.add(route);
     locales.add(locale);
@@ -1872,13 +1877,13 @@ function buildI18nMatrix(pages, options = {}) {
       grid.set(route, row);
     }
     if (!row.has(locale)) {
-      row.set(locale, { url, inspected: inspectedByUrl.has(url) });
+      row.set(locale, { url, inspected: inspectedByUrl.has(url), source: sourceOf(url) });
     }
   }
   for (const page of pages) {
     const url = new URL(page.fetch.finalUrl);
     const { route: selfRoute, locale: selfLocale } = splitRoute(url.pathname);
-    record(rowOf(page.fetch.finalUrl, selfRoute), selfLocale, page.fetch.finalUrl);
+    record(rowOf2(page.fetch.finalUrl, selfRoute), selfLocale, page.fetch.finalUrl);
     for (const alt of page.links.alternates) {
       let altUrl;
       try {
@@ -1888,7 +1893,7 @@ function buildI18nMatrix(pages, options = {}) {
       }
       const { route } = splitRoute(altUrl.pathname);
       const locale = alt.isXDefault ? "x-default" : localeIdentity(alt.hreflang);
-      record(rowOf(altUrl.toString(), route || selfRoute), locale, altUrl.toString());
+      record(rowOf2(altUrl.toString(), route || selfRoute), locale, altUrl.toString());
     }
   }
   for (const declared of options.declaredUrls ?? []) {
@@ -1899,7 +1904,7 @@ function buildI18nMatrix(pages, options = {}) {
       continue;
     }
     const { route, locale } = splitRoute(url.pathname);
-    record(rowOf(url.toString(), route), locale, url.toString());
+    record(rowOf2(url.toString(), route), locale, url.toString());
   }
   const sortedLocales = sortLocales([...locales]);
   const sortedRoutes = [...routes].sort((a, b) => a.localeCompare(b));
@@ -1908,11 +1913,20 @@ function buildI18nMatrix(pages, options = {}) {
     cells[route] = {};
     for (const locale of sortedLocales) {
       const slot = grid.get(route)?.get(locale);
-      cells[route][locale] = slot ? { url: slot.url, inspected: slot.inspected } : { url: null, inspected: false };
+      cells[route][locale] = slot ? { url: slot.url, inspected: slot.inspected, source: slot.source } : { url: null, inspected: false };
     }
   }
   return { locales: sortedLocales, routes: sortedRoutes, cells };
 }
+function countUnverifiedAlternates(matrix) {
+  let count2 = 0;
+  for (const route of matrix.routes) {
+    for (const locale of matrix.locales) {
+      if (matrix.cells[route]?.[locale]?.source === "alternate") count2++;
+    }
+  }
+  return count2;
+}
 function reciprocityIssues(pages) {
   const issues = [];
   const byUrl = /* @__PURE__ */ new Map();
@@ -2013,6 +2027,9 @@ function declaredLocales(page) {
   }
   return out;
 }
+function rowOf(site, url, pathname) {
+  return site.clusterRouteOf?.(url) ?? splitRoute(pathname).route;
+}
 function sitemapLocalesByRoute(site) {
   const byRoute = /* @__PURE__ */ new Map();
   for (const entry of site.discovery?.urls ?? []) {
@@ -2022,8 +2039,9 @@ function sitemapLocalesByRoute(site) {
     } catch {
       continue;
     }
-    const { route, locale } = splitRoute(pathname);
+    const { locale } = splitRoute(pathname);
     if (locale === "x-default") continue;
+    const route = rowOf(site, entry.loc, pathname);
     const set = byRoute.get(route) ?? /* @__PURE__ */ new Set();
     set.add(locale.toLowerCase());
     byRoute.set(route, set);
@@ -2087,7 +2105,7 @@ var hreflangSitemapMismatch = {
       } catch {
         continue;
       }
-      const { route } = splitRoute(pathname);
+      const route = rowOf(site, page.fetch.finalUrl, pathname);
       const inSitemap = bySitemap.get(route);
       if (!inSitemap || inSitemap.size === 0) continue;
       const onlyInSitemap = sorted(new Set([...inSitemap].filter((l) => !head.has(l))));
@@ -2588,39 +2606,140 @@ function normalise(raw) {
   return `${url.origin}${path}${url.search}`;
 }
 var X_DEFAULT = "x-default";
+function clusterCount(rows) {
+  return new Set(rows.values()).size;
+}
+function indexOf(rows, conflicts, refused) {
+  return {
+    rows,
+    routeOf: (url) => {
+      const key = normalise(url);
+      return key === null ? void 0 : rows.get(key);
+    },
+    size: clusterCount(rows),
+    conflicts,
+    refused
+  };
+}
 function buildClusterIndex(entries) {
-  const routeByUrl = /* @__PURE__ */ new Map();
+  const rows = /* @__PURE__ */ new Map();
   const conflicts = [];
-  const anchors = /* @__PURE__ */ new Set();
+  let refused = 0;
   for (const entry of entries) {
     const alternates = entry.alternates;
     if (!alternates || alternates.length === 0) continue;
     const xDefault = alternates.find((a) => a.hreflang.trim().toLowerCase() === X_DEFAULT);
-    if (!xDefault) continue;
+    if (!xDefault) {
+      refused++;
+      continue;
+    }
     const anchorUrl = normalise(xDefault.href);
-    if (!anchorUrl) continue;
+    if (!anchorUrl) {
+      refused++;
+      continue;
+    }
     const { route: anchorRoute } = splitRoute(new URL(anchorUrl).pathname);
-    anchors.add(anchorRoute);
     const members = [entry.loc, ...alternates.map((a) => a.href)];
     for (const member of members) {
       const url = normalise(member);
       if (!url) continue;
-      const existing = routeByUrl.get(url);
+      const existing = rows.get(url);
       if (existing === void 0) {
-        routeByUrl.set(url, anchorRoute);
+        rows.set(url, anchorRoute);
         continue;
       }
       if (existing !== anchorRoute && !conflicts.includes(url)) conflicts.push(url);
     }
   }
-  return {
-    routeOf: (url) => {
-      const key = normalise(url);
-      return key === null ? void 0 : routeByUrl.get(key);
-    },
-    size: anchors.size,
-    conflicts
+  return indexOf(rows, conflicts, refused);
+}
+function readHeads(pages) {
+  const seen = /* @__PURE__ */ new Map();
+  for (const page of pages) {
+    const self = normalise(page.fetch.finalUrl);
+    if (self === null || seen.has(self)) continue;
+    const peers = /* @__PURE__ */ new Set();
+    const xDefaults = /* @__PURE__ */ new Set();
+    for (const alt of page.links.alternates) {
+      const href = normalise(alt.href);
+      if (href === null) continue;
+      if (alt.isXDefault) xDefaults.add(href);
+      else if (href !== self) peers.add(href);
+    }
+    seen.set(self, { peers, xDefault: xDefaults.size === 1 ? [...xDefaults][0] : null });
+  }
+  return seen;
+}
+function buildHeadClusterIndex(pages) {
+  const seen = readHeads(pages);
+  const parent = /* @__PURE__ */ new Map();
+  for (const url of seen.keys()) parent.set(url, url);
+  const find = (start) => {
+    let root = start;
+    while (parent.get(root) !== root) root = parent.get(root);
+    let cursor = start;
+    while (parent.get(cursor) !== root) {
+      const next = parent.get(cursor);
+      parent.set(cursor, root);
+      cursor = next;
+    }
+    return root;
+  };
+  for (const [url, decl] of seen) {
+    for (const peer of decl.peers) {
+      if (!seen.get(peer)?.peers.has(url)) continue;
+      const [a, b] = [find(url), find(peer)];
+      if (a !== b) parent.set(a, b);
+    }
+  }
+  const components = /* @__PURE__ */ new Map();
+  for (const url of seen.keys()) {
+    const root = find(url);
+    const members = components.get(root);
+    if (members) members.push(url);
+    else components.set(root, [url]);
+  }
+  const rows = /* @__PURE__ */ new Map();
+  let refused = 0;
+  for (const members of components.values()) {
+    if (members.length < 2) continue;
+    const declared = new Set(
+      members.map((m) => seen.get(m)?.xDefault).filter((x) => Boolean(x))
+    );
+    if (declared.size !== 1) {
+      refused++;
+      continue;
+    }
+    const anchor = [...declared][0];
+    if (!members.includes(anchor)) {
+      refused++;
+      continue;
+    }
+    const { route } = splitRoute(new URL(anchor).pathname);
+    for (const member of members) rows.set(member, route);
+  }
+  return indexOf(rows, [], refused);
+}
+function combineClusterIndexes(primary, secondary) {
+  const rows = new Map(secondary.rows);
+  const conflicts = [...primary.conflicts];
+  const add = (url) => {
+    if (!conflicts.includes(url)) conflicts.push(url);
   };
+  for (const url of secondary.conflicts) add(url);
+  for (const [url, route] of primary.rows) {
+    const claimed = rows.get(url);
+    if (claimed !== void 0 && claimed !== route) add(url);
+    rows.set(url, route);
+  }
+  return indexOf(rows, conflicts, primary.refused + secondary.refused);
+}
+function clustersOnlyFrom(primary, secondary) {
+  const routes = /* @__PURE__ */ new Set();
+  for (const [url, route] of secondary.rows) {
+    if (!primary.rows.has(url)) routes.add(route);
+  }
+  return routes.size;
 }
 
 // src/lib/core/coverage.ts
@@ -3626,7 +3745,7 @@ async function runAudit(inputUrl, options = {}) {
     crawlFallback: false
   }).catch(() => void 0);
   const allSitemapUrls = (discovery?.urls ?? []).map((u) => u.loc);
-  const clusters = buildClusterIndex(discovery?.urls ?? []);
+  const sitemapClusters = buildClusterIndex(discovery?.urls ?? []);
   const coverageMode = options.coverage ?? (allSitemapUrls.length > 0 ? "structural" : "all");
   const selection = coverageMode === "structural" && allSitemapUrls.length > 0 ? selectByStructure(allSitemapUrls, { locales: options.locales }) : void 0;
   const sitemapUrls = selection?.urls ?? allSitemapUrls;
@@ -3729,6 +3848,9 @@ async function runAudit(inputUrl, options = {}) {
     );
   }
   if (!ICU_KNOWS_LANGUAGES) warnings.push(ICU_UNAVAILABLE_WARNING);
+  const headClusters = buildHeadClusterIndex(htmlPages);
+  const clusters = combineClusterIndexes(sitemapClusters, headClusters);
+  const clustersFromHead = clustersOnlyFrom(sitemapClusters, headClusters);
   const matrix = buildI18nMatrix(htmlPages, {
     declaredUrls: sitemapUrls,
     locales: localeAxis.locales,
@@ -3736,7 +3858,13 @@ async function runAudit(inputUrl, options = {}) {
   });
   if (clusters.conflicts.length > 0) {
     warnings.push(
-      `${clusters.conflicts.length} URL(s) are declared in two different hreflang clusters by the sitemap; the first declaration was kept: ${clusters.conflicts.slice(0, 3).join(", ")}${clusters.conflicts.length > 3 ? `, +${clusters.conflicts.length - 3} more` : ""}.`
+      `${clusters.conflicts.length} URL(s) are declared in two different hreflang clusters by the sitemap and/or the pages' \`<head>\`; the sitemap's declaration was kept: ${clusters.conflicts.slice(0, 3).join(", ")}${clusters.conflicts.length > 3 ? `, +${clusters.conflicts.length - 3} more` : ""}.`
+    );
+  }
+  const unverifiedAlternates = countUnverifiedAlternates(matrix);
+  if (unverifiedAlternates > 0) {
+    warnings.push(
+      `${unverifiedAlternates} translation${unverifiedAlternates === 1 ? "" : "s"} in the matrix ${unverifiedAlternates === 1 ? "is" : "are"} vouched for only by an \`hreflang\` on another page \u2014 not crawled, and not in the sitemap. They are counted as present, so a page advertised but not served reads as translated rather than missing.`
     );
   }
   const holes = localeAxis.multilingual ? deriveTranslationHoles(matrix, options.ignoreHoles) : [];
@@ -3757,7 +3885,11 @@ async function runAudit(inputUrl, options = {}) {
     matrix,
     localeAxis,
     discovery,
-    robots
+    robots,
+    // Same index the matrix was built from, so a rule that groups by route
+    // and the grid that reports holes cannot disagree about which URLs are
+    // one page.
+    clusterRouteOf: clusters.routeOf
   };
   const siteOccurrence = /* @__PURE__ */ new Map();
   const siteIssues = lintSite(siteContext).map((draft) => {
@@ -3883,13 +4015,22 @@ async function runAudit(inputUrl, options = {}) {
       },
       ...ignoredHoles > 0 ? { ignoredHoles } : {},
       ...duplicatePages > 0 ? { duplicatePages } : {},
+      ...unverifiedAlternates > 0 ? { unverifiedAlternates } : {},
       // Only when the site declared clusters. A merge that changes which rows
       // exist must be visible: the alternative is a hole count that moved for
       // a reason nothing in the report explains.
-      ...clusters.size > 0 ? {
+      ...clusters.size > 0 || clusters.refused > 0 ? {
         declaredClusters: {
           count: clusters.size,
-          ...clusters.conflicts.length > 0 ? { conflicts: clusters.conflicts.length } : {}
+          ...clusters.conflicts.length > 0 ? { conflicts: clusters.conflicts.length } : {},
+          // Which source did the work. On a site that declares both ways
+          // the two indexes describe the same clusters, so this counts the
+          // rows only the `<head>` supplied rather than a difference of
+          // totals that would claim a gain that is not there.
+          ...clustersFromHead > 0 ? { fromHead: clustersFromHead } : {},
+          // Declared and not used, for want of an anchor to name the row
+          // after. A declaration we saw and did not act on has to be said.
+          ...clusters.refused > 0 ? { refused: clusters.refused } : {}
         }
       } : {},
       ...discovery ? {
@@ -4566,4 +4707,4 @@ export {
   summarize,
   Logger
 };
-//# sourceMappingURL=chunk-UOKYUKHQ.js.map
+//# sourceMappingURL=chunk-SKX7UCGC.js.map