@redpanda-data/docs-extensions-and-macros 5.9.0 → 5.11.0

package/extensions/REFERENCE.adoc

@@ -485,6 +485,97 @@ antora:
     - require: '@redpanda-data/docs-extensions-and-macros/extensions/set-available-attachment-versions'
 ----
 
+== URL to xref
+
+This extension converts absolute docs.redpanda.com URLs found in page and partial content into Antora xrefs, using the content catalog as the single source of truth. A URL is converted only when it maps to a page that is actually published in the current build, so the extension never guesses at URL structure. Converted links are validated by Antora's own xref resolution, and internal URLs that match no published page are logged as warnings — which makes this extension a broken-internal-link detector for the whole site.
+
+Behavior details:
+
+* Runs at `contentClassified`, before AsciiDoc conversion, so emitted xrefs flow through Antora's normal resolution and broken-xref logging.
+* URLs inside listing, literal, fenced, and passthrough blocks, inline code spans, attribute entry lines, and macro attribute values (such as `link=` on an image macro) are left untouched.
+* Legacy URL shapes (`/docs/...`, `/current/...`, `/vX.Y/...`, and pre-rename component slugs such as `/redpanda-connect/`) are rewritten to candidate paths, but a candidate is only used when it matches a published page.
+* Fragments (`#anchor`) and link labels are preserved, including a label that a generator wrapped across a line break, whose line breaks are collapsed to single spaces.
+* An unlabeled URL becomes `xref:...[]` so that Antora fills in the target page title. An unlabeled URL that carries a fragment is given link text explicitly, because Antora cannot resolve a section title and would render the raw resource ID as the link text: the extension uses the heading the fragment points at, falls back to the target page title, and leaves the URL as a raw link if it can read neither. Heading ids are also read from the partials a target page includes, because the reference pages that hold most anchor targets are thin wrappers around generated partials.
+* Targets in the latest version of a component emit an unversioned xref; targets pinned to an older version emit `version@component:module:page`.
+* A URL that points at a page's former path resolves through the `page-aliases` attribute that the page declares. Antora does not register those aliases as catalog files until it converts documents, which is after this extension runs, so the extension reads them from the page header. Only aliases in the same component and version as the page that declares them are mapped.
+* A URL that carries a query string is left as a raw link, because an xref cannot express one and dropping it would change the link (for example `?platform=kubernetes`, which the UI reads to preselect a tab). These are reported at info level, not as warnings.
+* Builds that publish the latest version under its real version number instead of a symbolic segment (such as this repository's preview build) still resolve `/current/...` URLs, so the same content works in both.
+
+=== Environment variables
+
+This extension does not require any environment variables.
+
+=== Configuration options
+
+hostnames (optional)::
+List of hostnames treated as internal documentation URLs. Default: `['docs.redpanda.com']`.
+
+ignore (optional)::
+List of regular expressions matched against URL paths. Matching URLs are left untouched with no warning. Use this for paths on the docs domain that live outside the Antora catalog. Default: `['^/api/', '^/mcp(/|$)']` (the API reference is hosted by Bump.sh, and `/mcp` is the docs MCP server endpoint).
+
+log_unconverted (optional)::
+Whether to log a warning for each internal URL that matches no published page. Default: `true`.
+
+=== Registration
+
+Register the `url-to-xref` extension in the Antora playbook under the `antora.extensions` key like so:
+
+[source,yaml]
+----
+antora:
+  extensions:
+    - require: '@redpanda-data/docs-extensions-and-macros/extensions/url-to-xref'
+----
+
+== External link checker
+
+This extension collects every external http(s) URL referenced in page and partial content and verifies that it responds, reporting dead links in the build log. Internal links are covered by the url-to-xref extension plus Antora's own xref validation; this extension covers everything that points off-site.
+
+Because it performs network requests, register it only in builds where that belongs (for example a scheduled nightly build), not necessarily on every preview build.
+
+Responses are classified as follows:
+
+* 2xx/3xx: ok
+* 404/410: broken — logged as a warning, or at error level with `fail_on_broken`
+* 401/403/429: unverifiable — logged at info level, because bot-walled or rate-limited sites are not broken
+* Other statuses, network errors, and timeouts (after one retry): unverifiable — logged as a warning
+
+=== Environment variables
+
+This extension does not require any environment variables.
+
+=== Configuration options
+
+internal_hostnames (optional)::
+Hostnames to skip because they are internal documentation URLs. Default: `['docs.redpanda.com']`.
+
+include (optional)::
+List of regular expressions. When set, only URLs matching at least one pattern are checked.
+
+exclude (optional)::
+List of regular expressions. URLs matching any pattern are skipped.
+
+concurrency (optional)::
+Number of simultaneous requests. Default: `10`.
+
+timeout (optional)::
+Per-request timeout in milliseconds. Default: `10000`.
+
+fail_on_broken (optional)::
+Log broken links at error level so a build run with `--log-failure-level=error` fails. Default: `false`.
+
+=== Registration
+
+Register the `external-link-checker` extension in the Antora playbook under the `antora.extensions` key like so:
+
+[source,yaml]
+----
+antora:
+  extensions:
+    - require: '@redpanda-data/docs-extensions-and-macros/extensions/external-link-checker'
+      fail_on_broken: false
+----
+
 == Validate attributes
 
 This extension ensures the consistency and validity of page attributes, focusing on validating page categories against a predefined list of valid categories and subcategories. It automatically adds missing parent categories for any specified subcategories and removes any specified categories that are invalid. Additionally, it processes specific environment attributes, setting corresponding page-level attributes when environment conditions are met.

package/extensions/external-link-checker.js

@@ -0,0 +1,132 @@
+'use strict'
+
+const { scanContentUrls } = require('./util/scan-content-urls')
+const { raiseListenerLimit } = require('./util/raise-listener-limit')
+
+/**
+ * External Link Checker Extension
+ *
+ * Collects every external http(s) URL referenced in page and partial content
+ * and verifies it responds, reporting dead links in the build log. Internal
+ * links are covered by the url-to-xref extension plus Antora's own xref
+ * validation; this extension covers everything that points off-site.
+ *
+ * Because it performs network requests, register it only where that belongs
+ * (for example a nightly build), not necessarily on every preview build:
+ *
+ * antora:
+ *   extensions:
+ *     - require: '@redpanda-data/docs-extensions-and-macros/extensions/external-link-checker'
+ *       # internal_hostnames: ['docs.redpanda.com']  hosts to skip (default)
+ *       # include: []          regex allowlist; when set, only matching URLs are checked
+ *       # exclude: []          regex denylist
+ *       # concurrency: 10      simultaneous requests
+ *       # timeout: 10000       per-request timeout in ms
+ *       # fail_on_broken: false  log broken links at error level so a build
+ *       #                        run with --log-failure-level=error fails
+ *
+ * Classification:
+ * - 2xx/3xx: ok
+ * - 404/410: broken (warn, or error with fail_on_broken)
+ * - 401/403/429: unverifiable (info) — bot-walled or rate-limited, not broken
+ * - other statuses, network errors, timeouts (after one retry): unverifiable (warn)
+ */
+
+const USER_AGENT = 'redpanda-docs-link-checker'
+
+module.exports.register = function ({ config = {} }) {
+  raiseListenerLimit(this)
+  const logger = this.getLogger('external-link-checker-extension')
+  const internalHostnames = new Set(config.internalHostnames || ['docs.redpanda.com'])
+  const include = (config.include || []).map((pattern) => new RegExp(pattern))
+  const exclude = (config.exclude || []).map((pattern) => new RegExp(pattern))
+  const concurrency = config.concurrency || 10
+  const timeout = config.timeout || 10000
+  const failOnBroken = Boolean(config.failOnBroken)
+
+  this.on('contentClassified', async ({ contentCatalog }) => {
+    const urlReferences = collectExternalUrls(contentCatalog, { internalHostnames, include, exclude })
+    if (!urlReferences.size) return
+    const counts = { ok: 0, broken: 0, unverifiable: 0 }
+    await runWithConcurrency([...urlReferences.keys()], concurrency, async (url) => {
+      const verdict = await checkUrl(url, { timeout })
+      counts[verdict.classification]++
+      const pages = [...urlReferences.get(url)].join(', ')
+      if (verdict.classification === 'broken') {
+        const message = `Broken external link ${url} (HTTP ${verdict.status}) referenced in: ${pages}`
+        failOnBroken ? logger.error(message) : logger.warn(message)
+      } else if (verdict.classification === 'unverifiable') {
+        const reason = verdict.status ? `HTTP ${verdict.status}` : verdict.reason
+        const log = verdict.status && [401, 403, 429].includes(verdict.status) ? 'info' : 'warn'
+        logger[log](`Could not verify external link ${url} (${reason}) referenced in: ${pages}`)
+      }
+    })
+    logger.info(
+      `Checked ${urlReferences.size} external links: ${counts.ok} ok, ${counts.broken} broken, ` +
+        `${counts.unverifiable} unverifiable`
+    )
+  })
+}
+
+function collectExternalUrls (contentCatalog, { internalHostnames, include, exclude }) {
+  const urlReferences = new Map()
+  const files = contentCatalog
+    .getPages((page) => page.out)
+    .concat(contentCatalog.findBy({ family: 'partial' }))
+  for (const file of files) {
+    if (!file.contents) continue
+    for (const match of scanContentUrls(file.contents.toString())) {
+      let url
+      try {
+        url = new URL(match.url)
+      } catch {
+        continue
+      }
+      if (internalHostnames.has(url.hostname)) continue
+      if (include.length && !include.some((pattern) => pattern.test(match.url))) continue
+      if (exclude.some((pattern) => pattern.test(match.url))) continue
+      if (!urlReferences.has(match.url)) urlReferences.set(match.url, new Set())
+      urlReferences.get(match.url).add(file.path)
+    }
+  }
+  return urlReferences
+}
+
+async function checkUrl (url, { timeout, fetchFn = module.exports._fetch }) {
+  let lastError
+  for (const attempt of [{ method: 'HEAD' }, { method: 'GET' }, { method: 'GET', retry: true }]) {
+    try {
+      const response = await fetchFn(url, {
+        method: attempt.method,
+        redirect: 'follow',
+        headers: { 'user-agent': USER_AGENT },
+        signal: AbortSignal.timeout(timeout),
+      })
+      const status = response.status
+      // HEAD rejected or unsupported: many CDNs and bot walls refuse HEAD
+      // (403, 404, ...) but answer GET normally, so confirm any HEAD failure
+      // with GET before classifying
+      if (attempt.method === 'HEAD' && status >= 400) continue
+      if (status < 400) return { classification: 'ok', status }
+      if (status === 404 || status === 410) return { classification: 'broken', status }
+      return { classification: 'unverifiable', status }
+    } catch (error) {
+      lastError = error
+      if (attempt.retry) break
+    }
+  }
+  return { classification: 'unverifiable', status: null, reason: lastError ? lastError.message : 'request failed' }
+}
+
+async function runWithConcurrency (items, limit, task) {
+  const queue = [...items]
+  const workers = Array.from({ length: Math.min(limit, queue.length) }, async () => {
+    while (queue.length) await task(queue.shift())
+  })
+  await Promise.all(workers)
+}
+
+// Indirection so tests can stub the HTTP layer without a network.
+module.exports._fetch = (...args) => fetch(...args)
+module.exports.checkUrl = checkUrl
+module.exports.collectExternalUrls = collectExternalUrls