@redpanda-data/docs-extensions-and-macros 5.55.0 → 5.56.0

package/extensions/REFERENCE.adoc

@@ -100,6 +100,29 @@ This extension generates an Algolia index for each version of each component. Th
 
 NOTE: Only pages that include an `<article>` element with the `doc` class are indexed.
 
+When the build has the docs taxonomy (see <<Docs taxonomy>>), every Product Docs record also carries these attributes, next to `categories`:
+
+[cols="1,3"]
+|===
+|Attribute |Value
+
+|`topics`, `topicIds`
+|The topic concepts of the page's authored `page-categories`, with the broader concepts they imply (Replication implies High Availability). Labels and IDs.
+
+|`useCases`, `useCaseIds`
+|The use-case concepts of the page's categories.
+
+|`technologies`, `technologyIds`
+|The technology concepts of the page's categories, those a topic implies (Iceberg Topics implies Apache Iceberg), and the technology whose `component` is the page's component (a page in `connect` is about Redpanda Connect).
+
+|`deployment`, `deploymentIds`
+|The deployment concept of the page's `env-*` attributes, and its broader concept.
+|===
+
+`categories` is unchanged: it is still the legacy list that validate-attributes writes, because labs-home filters on it. Labs records get none of the new attributes, and keep their string `deployment`.
+
+The Algolia index settings are not in any repository. To filter or facet on the new attributes, add them to `attributesForFaceting` in the index settings.
+
 === Environment variables
 
 - `ALGOLIA_ADMIN_API_KEY` (required)
@@ -730,6 +753,17 @@ This extension does not require any environment variables.
 
 There are no configurable options for this extension. It operates based on site attributes defined in `add-global-attributes.js` to determine valid categories and subcategories.
 
+=== With the docs taxonomy
+
+When the build has the docs taxonomy (see <<Docs taxonomy>>), each `page-categories` value is also resolved through it:
+
+* A value from `valid-categories.yml` passes through unchanged, so `page-categories` is exactly what it was without the taxonomy. When the value is a deprecated concept, the build logs one warning per value, with the number of pages, a sample, and the replacement. Labs and solutions pages are left to the legacy check: labs is retiring, and solutions-catalog resolves solution categories itself.
+* A value only the taxonomy knows, such as a new label like `Iceberg Topics`, is replaced by the legacy category of the concept it resolves to, or dropped when there is none. It is not reported as invalid.
+* A value neither knows warns per page and is dropped, as before.
+* A legacy value the taxonomy does not resolve warns as a gap in `taxonomy.yml`.
+
+Nothing here fails the build. The authored list is kept in `page-categories-authored` for the extensions that resolve concepts from it.
+
 === Registration
 
 Register the `validate-attributes` extension in the Antora playbook under the `antora.extensions` key like so:
@@ -787,7 +821,26 @@ antora:
     - require: '@redpanda-data/docs-extensions-and-macros/extensions/related-labs-extension.js'
 ----
 
-== Global attributes
+== Docs taxonomy
+
+`extension-utils/taxonomy.js` loads the docs taxonomy, the controlled vocabulary in the docs repository at `shared/modules/ROOT/attachments/taxonomy.yml`, and validates it against `taxonomy.schema.json` next to it. It is not an extension to register: validate-attributes and the Algolia indexer load it on first use and share it on `siteCatalog.taxonomy`.
+
+The file is read from the content catalog as an attachment of the `shared` component, so it is never merged into page attributes. <<Global attributes>> only reads `modules/ROOT/partials/`.
+
+The file has seven concept schemes (topic, task, content-type, use-case, industry, technology, deployment). Every concept has an `id`, a `prefLabel`, `altLabels`, a `definition`, a `broader` concept, `related` concepts, and `deprecated` with `replacedBy`. Every old category is either the label of a live concept or a deprecated concept whose `replacedBy` names its replacement.
+
+At load, the build fails when the file does not match the schema, when an ID repeats, when a label or altLabel repeats in any scheme (case-insensitive), or when a `broader`, `related`, `implies`, or `replacedBy` ID does not exist. When the file is absent, the build logs one warning and every consumer keeps its behavior without the taxonomy, so the docs repository and this package can deploy in either order. When the schema is absent, only the structural checks run.
+
+`Taxonomy#resolve(value)` returns the concept a label, altLabel, or ID names, and for a deprecated concept its `replacedBy` IDs. `Taxonomy#expand(ids)` adds the broader concepts and `implies` targets as implied. `pageFacets` turns a page's authored categories, deployment type, and component into the search facets.
+
+Outside Antora, `loadTaxonomyFrom({ source, root })` loads the same file from a path or URL, or else from the first that exists of the docs checkout at `root`, the docs checkout at `$REDPANDA_DOCS_PATH`, and the published copy at https://docs.redpanda.com/shared/_attachments/taxonomy.yml. It validates against the schema beside whichever it used.
+
+The metadata rules live in `extension-utils/metadata-check.js` (`checkPageMetadata`). validate-attributes runs them on Product Docs pages, solutions-catalog on solution overviews, and `doc-tools check-metadata` on page headers in a pull request, so the build and the PR check cannot disagree. See `check-metadata` in the CLI reference for the rules and their severities, and `.github/workflows/check-metadata.yml` for the reusable PR check.
+
+Suggestions live in `extension-utils/metadata-suggest.js` (`suggestForPage`), which `doc-tools check-metadata --suggest` runs. It matches every live concept's labels against a page's title, description, path, headings, and body, deterministically and with no model, and returns up to a few concepts per facet that the page does not carry yet, each with a score and the evidence for it. It honors the facet cardinalities in the taxonomy, `implies`, and `broader`, and never suggests a deprecated concept. Another tool that tags content against the taxonomy should call it rather than match labels itself, so every tagger agrees.
+
+`extension-utils/metadata-candidates.js` finds the other side: frequent, distinctive terms on a page that no concept's label matches (`pageCandidates`), and the same across a repository with the pages each is on (`aggregateCandidates`). `doc-tools check-metadata --suggest` lists them per page and `--candidates-only` aggregates them. They are evidence for proposing a new concept or altLabel, deterministic like the suggestions.
+
 
 This extension collects Asciidoc attributes from the https://github.com/redpanda-data/docs-site[`shared` component] or a local YAML file and makes them available to all component versions. Having global attributes is useful for consistent configuration of local and production builds.
 
@@ -1300,8 +1353,38 @@ The extension exports these attributes to YAML frontmatter:
 - `page-categories`: Categories
 - `page-topic-type`: Topic type (tutorial, how-to, concept, reference)
 - `page-personas`: Target audience
+- `page-generated-content`: Always present: `none`, `partial` (the page includes generated files) or `all` (the page itself was generated)
+- `page-autogenerated`, `page-generated-by`, `page-generated-from`: Set when the page itself was written by a generator: the tool that wrote it, and the source it names (`page-generated-from` only when the marker names one)
+- `page-generated-sources`: One entry per include whose file was written by a generator: the include target, the generator, its version when the marker carries one, and the source it names
+- `page-introduced-in`, `page-introduced-in-product`: Release that introduced the page's feature, as a bare `X.Y.Z`, and the product it belongs to (generated Connect reference pages, self-managed only)
 - Custom attributes from allowlist
 
+=== Generated content provenance
+
+Every doc-tools generator writes one comment line near the top of each file it produces, built by `cli-utils/generated-marker.js`:
+
+----
+// This content is autogenerated. Do not edit manually. Generated by doc-tools (<generator>) from <source>. <how to change it>
+----
+
+The marker names no doc-tools version. Generated files are committed, so a version would rewrite the first line of every generated file on each release even when nothing else changed. A marker that does carry a token after `doc-tools` (`doc-tools 5.55.0 (rpk-docs)`) is still read, and the token is reported as the version when it is one. Pages generated during the site build, which are never committed, record the version of the package that built them.
+
+While the catalog still holds AsciiDoc (`contentClassified`), the extension reads that marker on every page and on every file the page includes, then records what it found as page attributes (`documentsConverted`) so both the Markdown twin and UI templates can use it. Every twin states `page-generated-content`, so `none` is a statement and not an absence. A page that is generated end to end gets `all`, `page-autogenerated: true`, `page-generated-by` and `page-generated-from`; a hand-written page that includes generated partials gets `partial` and `page-generated-sources`, for example:
+
+[,yaml]
+----
+page-generated-content: partial
+page-generated-sources:
+  - include: connect:components:partial$descriptions/inputs/postgres_cdc.adoc
+    generator: rpcn-connector-docs
+    version: null
+    source: the component spec in redpanda-data/connect
+----
+
+Older files that carry only `// This content is autogenerated. Do not edit manually.` are still recognized; they report a null generator and source, and the twin has no `page-generated-from`. A source is read only right after `doc-tools (<generator>)` or after `autogenerated`, so a later "from" in a legacy banner's prose is not taken for one. A comment counts as a marker only when it opens with `This content is autogenerated`, `Autogenerated`, `AUTO-GENERATED`, or `Generated by doc-tools`, so a hand-written comment such as `// Generated by running rpk cluster info` does not. The partials that ship in the Connect release asset carry connect's own banner, which names the file that registers the component as `<repository>[@<version>]:<path>` (for example `// This content is autogenerated from the component spec in redpanda-data/benthos@v4.84.0:internal/impl/pure/processor_mapping.go. Do not edit manually.`), or the repository when the generator could not read the name; either reports that source and a null generator. An extension that creates pages at build time sets `page.generatedProvenance` on the file itself, which this scan honors.
+
+Include detection is best effort. Each `include::` target at the start of a line is resolved as a partial in the page's context (falling back to pages and examples), so a page-relative include such as `include::shared.adoc[]` is looked up as `partial$shared.adoc`. Includes inside comment blocks or inactive `ifdef` branches are counted, targets with an attribute reference are skipped, and nested includes (a hand-written partial that includes a generated one) are not followed.
+
 === Example output
 
 [,yaml]
@@ -1396,7 +1479,8 @@ The `convert-sitemap-to-markdown` extension generates human-readable markdown ve
 - Generates individual `.md` files for each sitemap
 - Creates master `sitemap-all.md` combining all pages
 - Organizes URLs by component/version
-- Includes page metadata (modified dates, priority)
+- Uses each page's document title and `:description:`, falling back to a title from the URL for files that aren't Antora pages
+- Includes page metadata (modified dates, priority) on a nested line under each entry, so the entry line keeps the `- [Title](url): description` form
 - Uses Antora content catalog (no filesystem operations)
 
 === Configuration
@@ -1440,8 +1524,10 @@ Total pages: 4,126
 
 ### Current
 
-- [Get Started](https://docs.redpanda.com/current/get-started/) (modified: 2024-03-29)
-- [Deploy](https://docs.redpanda.com/current/deploy/) (modified: 2024-03-28)
+- [Introduction to Redpanda](https://docs.redpanda.com/current/get-started/): Learn about Redpanda and its features.
+  - modified: 2024-03-29
+- [Deploy](https://docs.redpanda.com/current/deploy/)
+  - modified: 2024-03-28
 ----
 
 === Use cases

package/extensions/algolia-indexer/generate-index.js

@@ -3,6 +3,7 @@
 const { parse } = require('node-html-parser')
 const { decode } = require('html-entities')
 const path = require('path')
+const { pageFacets, authoredCategories } = require('../../extension-utils/taxonomy')
 
 // Create encoder once at module scope for efficiency
 const textEncoder = new TextEncoder()
@@ -20,10 +21,12 @@ const textEncoder = new TextEncoder()
  * @param {Boolean} config.indexLatestOnly - If true, only index the latest version of any given page.
  * @param {Array} config.excludes - CSS selectors for elements to exclude from indexing.
  * @param {Object} config.logger - Logger to use
+ * @param {Object} [config.taxonomy] - The site taxonomy (extension-utils/taxonomy.js). When
+ *   set, Product Docs records also carry topics, useCases, technologies, and deployment.
  * @typedef {Object} SearchIndexData
  * @returns {SearchIndexData} A data object that contains the Algolia index
  */
-function generateIndex (playbook, contentCatalog, { indexLatestOnly = false, excludes = [], logger } = {}) {
+function generateIndex (playbook, contentCatalog, { indexLatestOnly = false, excludes = [], logger, taxonomy } = {}) {
   // Use provided logger or create a no-op logger for tests
   if (!logger) {
     logger = process.env.NODE_ENV === 'test'
@@ -288,6 +291,10 @@ function generateIndex (playbook, contentCatalog, { indexLatestOnly = false, exc
       ? page.asciidoc.attributes['page-categories'].split(',').map(category => category.trim())
       : []
 
+    const facets = taxonomy && component.name !== 'labs'
+      ? pageFacets(taxonomy, { categories: authoredCategories(page.asciidoc?.attributes), deployment, component: component.name })
+      : null
+
     const commercialNames = page.asciidoc?.attributes['page-commercial-names']
       ? page.asciidoc.attributes['page-commercial-names'].split(',').map(name => name.trim())
       : []
@@ -332,6 +339,11 @@ function generateIndex (playbook, contentCatalog, { indexLatestOnly = false, exc
         indexItem.breadcrumbs = breadcrumbs
         indexItem.type = 'Doc'
         indexItem._tags = Array.isArray(tag) ? tag : [tag]
+        // Taxonomy facets, next to `categories` rather than instead of it:
+        // labs-home filters on `categories`, so it stays the legacy list until
+        // every consumer has moved. Arrays of prefLabels for display and
+        // faceting, plus the concept IDs, which do not change when a label does.
+        if (taxonomy) Object.assign(indexItem, facets)
       } else {
         indexItem.deployment = deployment
         indexItem.type = 'Lab'

package/extensions/algolia-indexer/index.js

@@ -1,6 +1,7 @@
 'use strict'
 
 const generateIndex = require('./generate-index')
+const { getTaxonomy } = require('../../extension-utils/taxonomy')
 const { raiseListenerLimit } = require('../util/raise-listener-limit')
 const algoliasearch = require('algoliasearch')
 const http = require('http')
@@ -51,8 +52,9 @@ function register ({
   })
   const index = client.initIndex(process.env.ALGOLIA_INDEX_NAME)
 
-  this.on('beforePublish', async ({ playbook, contentCatalog }) => {
-    const algolia = generateIndex(playbook, contentCatalog, { indexLatestOnly, excludes, logger })
+  this.on('beforePublish', async ({ playbook, contentCatalog, siteCatalog }) => {
+    const taxonomy = getTaxonomy({ contentCatalog, siteCatalog, logger })
+    const algolia = generateIndex(playbook, contentCatalog, { indexLatestOnly, excludes, logger, taxonomy })
 
     if (!algolia || Object.keys(algolia).length === 0) {
       logger.warn('No content to index for Algolia')

package/extensions/convert-sitemap-to-markdown.js

@@ -14,12 +14,14 @@
 
 const path = require('path')
 const { parseStringPromise } = require('xml2js')
+const { decode } = require('html-entities')
+const { toMarkdownUrl } = require('../extension-utils/url-utils')
 
 module.exports.register = function ({ config }) {
   const logger = this.getLogger('convert-sitemap-to-markdown')
 
   this
-    .on('beforePublish', async ({ playbook, siteCatalog }) => {
+    .on('beforePublish', async ({ playbook, siteCatalog, contentCatalog }) => {
       const startTime = Date.now()
 
       logger.info('Sitemap to markdown converter starting...')
@@ -37,10 +39,12 @@ module.exports.register = function ({ config }) {
 
         logger.info(`Found ${sitemapFiles.length} sitemap file(s)`)
 
+        const pageInfo = collectPageInfo(contentCatalog)
+
         // Convert each sitemap and collect all URLs
         const allUrls = []
         for (const sitemapFile of sitemapFiles) {
-          const urls = await convertSitemapToMarkdown(sitemapFile, siteCatalog, logger)
+          const urls = await convertSitemapToMarkdown(sitemapFile, siteCatalog, logger, pageInfo)
           if (urls) {
             allUrls.push(...urls)
           }
@@ -48,7 +52,7 @@ module.exports.register = function ({ config }) {
 
         // Create combined master sitemap if we have multiple sitemaps
         if (sitemapFiles.length > 1 && allUrls.length > 0) {
-          await createMasterSitemap(sitemapFiles, allUrls, siteCatalog, logger)
+          await createMasterSitemap(sitemapFiles, allUrls, siteCatalog, logger, pageInfo)
         }
 
         const duration = ((Date.now() - startTime) / 1000).toFixed(2)
@@ -63,7 +67,7 @@ module.exports.register = function ({ config }) {
 /**
  * Create a master combined sitemap from all individual sitemaps
  */
-async function createMasterSitemap(sitemapFiles, allUrls, siteCatalog, logger) {
+async function createMasterSitemap(sitemapFiles, allUrls, siteCatalog, logger, pageInfo) {
   const sitemapCount = sitemapFiles.length
   const sitemapWord = sitemapCount === 1 ? 'sitemap' : 'sitemaps'
 
@@ -81,7 +85,7 @@ async function createMasterSitemap(sitemapFiles, allUrls, siteCatalog, logger) {
 
   // Add all URLs grouped by component
   if (allUrls.length > 0) {
-    markdown += convertUrlsetToMarkdown(allUrls)
+    markdown += convertUrlsetToMarkdown(allUrls, pageInfo)
   }
 
   // Add to site catalog
@@ -97,7 +101,7 @@ async function createMasterSitemap(sitemapFiles, allUrls, siteCatalog, logger) {
  * Convert a sitemap file to markdown
  * Returns the URLs found in this sitemap for aggregation
  */
-async function convertSitemapToMarkdown(sitemapFile, siteCatalog, logger) {
+async function convertSitemapToMarkdown(sitemapFile, siteCatalog, logger, pageInfo) {
   const xmlContent = sitemapFile.contents.toString('utf8')
   const basename = path.basename(sitemapFile.out.path)
   const outputPath = basename.replace(/\.xml$/, '.md')
@@ -113,7 +117,7 @@ async function convertSitemapToMarkdown(sitemapFile, siteCatalog, logger) {
     // Handle standard sitemap
     if (parsed.urlset && parsed.urlset.url) {
       urls = parsed.urlset.url
-      markdown += convertUrlsetToMarkdown(urls)
+      markdown += convertUrlsetToMarkdown(urls, pageInfo)
     }
 
     // Handle sitemap index
@@ -139,7 +143,7 @@ async function convertSitemapToMarkdown(sitemapFile, siteCatalog, logger) {
 /**
  * Convert URL entries to markdown
  */
-function convertUrlsetToMarkdown(urls) {
+function convertUrlsetToMarkdown(urls, pageInfo = new Map()) {
   let markdown = '## Pages\n\n'
   markdown += `Total pages: ${urls.length.toLocaleString()}\n\n`
 
@@ -156,18 +160,23 @@ function convertUrlsetToMarkdown(urls) {
       const priority = url.priority ? url.priority[0] : ''
 
       if (loc) {
-        markdown += `- [${extractPageTitle(loc)}](${loc})`
+        const page = pageInfo.get(new URL(loc).pathname)
+        const title = (page && page.title) || extractPageTitle(loc)
+        markdown += `- [${title}](${loc})`
+        if (page && page.description) markdown += `: ${page.description}`
+        markdown += '\n'
 
+        // Metadata goes on a nested line. Readers of the llms.txt link form
+        // take everything after `: ` as the description, so metadata on the
+        // entry line would become part of it.
         const metadata = []
         if (lastmod) metadata.push(`modified: ${lastmod}`)
         if (changefreq) metadata.push(`frequency: ${changefreq}`)
         if (priority) metadata.push(`priority: ${priority}`)
 
         if (metadata.length > 0) {
-          markdown += ` (${metadata.join(', ')})`
+          markdown += `  - ${metadata.join(', ')}\n`
         }
-
-        markdown += '\n'
       }
     }
 
@@ -177,6 +186,32 @@ function convertUrlsetToMarkdown(urls) {
   return markdown
 }
 
+/**
+ * Map each published page's HTML and Markdown paths to its document title and
+ * description, so sitemap entries carry the same text as the page itself
+ * rather than a title guessed from the URL.
+ */
+function collectPageInfo(contentCatalog) {
+  const info = new Map()
+  if (!contentCatalog) return info
+  for (const page of contentCatalog.getPages((p) => p.pub && p.pub.url && p.out)) {
+    const asciidoc = page.asciidoc || {}
+    const title = plainText(asciidoc.doctitle || asciidoc.navtitle)
+    const description = plainText(asciidoc.attributes && asciidoc.attributes.description)
+    if (!title && !description) continue
+    const entry = { title: title.replace(/([[\]])/g, '\\$1'), description }
+    info.set(page.pub.url, entry)
+    info.set(toMarkdownUrl(page.pub.url), entry)
+  }
+  return info
+}
+
+// Converted titles and descriptions are HTML. Strip tags before decoding so an
+// escaped placeholder such as `&lt;name&gt;` survives as `<name>`.
+function plainText(html) {
+  return decode(String(html || '').replace(/<[^>]+>/g, '')).replace(/\s+/g, ' ').trim()
+}
+
 /**
  * Convert sitemap index to markdown
  */

package/extensions/convert-to-markdown.js

@@ -8,6 +8,98 @@ const TurndownService = require('turndown')
 const turndownPluginGfm = require('turndown-plugin-gfm')
 const { gfm } = turndownPluginGfm
 
+// Every doc-tools generator writes one comment line near the top of each file
+// it produces, in this shape:
+//
+//   // This content is autogenerated. Do not edit manually. Generated by doc-tools (<generator>) from <source>. <how to change it>
+//
+// (cli-utils/generated-marker.js builds it.) Some files written before the
+// version was dropped carry one after "doc-tools". Older files carry only the
+// first two sentences. The twin's front matter reports what the marker says, so a
+// reader (or an agent) can tell generated text from hand-written text on the
+// same page and knows where the fix goes. The comment must open with the
+// generated phrase, which every marker in the docs repos does ("This content
+// is autogenerated", "AUTO-GENERATED FILE", "Generated by `doc-tools ...`"),
+// so a prose comment such as "// Table generated by the build" is not one.
+// "Generated by" counts only when doc-tools follows, so a hand-written
+// "// Generated by running rpk cluster info, then edited" is not one either.
+const GENERATED_MARKER_SCAN_LINES = 15
+const GENERATED_MARKER_RE = /^\/\/\s*((?:this content is auto-?generated\b|auto-?generated\b|generated by\s+`?doc-tools\b).*)$/i
+// "doc-tools (property-docs)", or with a version token: "doc-tools 5.55.0
+// (rpk-docs)". A token that is not a version (the Python helpers once wrote
+// "unknown") still yields the generator.
+const MARKER_TOOL_RE = /doc-tools(?:\s+([^\s()]+))?\s*\(([^)]+)\)/i
+// The source is named only right after the tool, or after "autogenerated"
+// (connect's own banner), so a "from" later in a legacy banner's prose is not
+// read as one.
+const MARKER_SOURCE_RE = /(?:doc-tools(?:\s+[^\s()]+)?\s*\([^)]+\)|auto-?generated)\s+from\s+(.+?)(?:\.(?=\s|$)|$)/i
+const INCLUDE_TARGET_RE = /^include::([^[\s{]+)\[/gm
+
+/**
+ * Read the generated-content marker from the first lines of an AsciiDoc file.
+ * @param {Buffer|string} contents - The file's AsciiDoc source
+ * @returns {{note: string, generator: (string|null), version: (string|null), source: (string|null)}|null}
+ */
+function readGeneratedMarker (contents) {
+  if (!contents) return null
+  const text = Buffer.isBuffer(contents) ? contents.toString('utf8') : String(contents)
+  const lines = text.split(/\r?\n/, GENERATED_MARKER_SCAN_LINES)
+  for (const line of lines) {
+    const m = line.match(GENERATED_MARKER_RE)
+    if (!m) continue
+    const note = m[1].trim()
+    const tool = note.match(MARKER_TOOL_RE)
+    const generator = tool ? tool[2].trim() : null
+    const token = tool && tool[1] ? tool[1].replace(/^v(?=\d)/i, '') : null
+    const version = token && /^\d/.test(token) ? token : null
+    const source = (note.match(MARKER_SOURCE_RE) || [])[1] || null
+    return { note, generator, version, source: source ? source.trim() : null }
+  }
+  return null
+}
+
+/**
+ * List the include targets in an AsciiDoc source. Targets that carry an
+ * attribute reference cannot be resolved here and are skipped.
+ * @param {Buffer|string} contents - The file's AsciiDoc source
+ * @returns {string[]} Unique include targets, in order of first appearance
+ */
+function findIncludeTargets (contents) {
+  if (!contents) return []
+  const text = Buffer.isBuffer(contents) ? contents.toString('utf8') : String(contents)
+  const targets = []
+  for (const m of text.matchAll(INCLUDE_TARGET_RE)) {
+    if (!targets.includes(m[1])) targets.push(m[1])
+  }
+  return targets
+}
+
+/**
+ * Work out which text on a page is generated: the page itself, and any
+ * include whose file carries the marker. Runs while the catalog still holds
+ * AsciiDoc, before conversion.
+ * @param {Object} page - Antora page file (src, contents)
+ * @param {Object} contentCatalog - Antora content catalog
+ * @returns {{page: (Object|null), includes: Object[]}}
+ */
+function collectGeneratedProvenance (page, contentCatalog) {
+  const own = readGeneratedMarker(page.contents)
+  const includes = []
+  for (const target of findIncludeTargets(page.contents)) {
+    let resource
+    try {
+      resource = contentCatalog.resolveResource(target, page.src, 'partial', ['partial', 'page', 'example'])
+    } catch (err) {
+      resource = null
+    }
+    if (!resource || !resource.contents) continue
+    const marker = readGeneratedMarker(resource.contents)
+    if (!marker) continue
+    includes.push({ include: target, generator: marker.generator, version: marker.version, source: marker.source, note: marker.note })
+  }
+  return { page: own, includes }
+}
+
 /**
  * Decode HTML entities in a string
  * @param {string} str - String with potential HTML entities
@@ -70,6 +162,17 @@ function generateFrontmatter(page) {
     'page-eol-date',
     'page-git-created-date',
     'page-git-modified-date',
+    // Generated-content provenance, from the markers generators write
+    'page-generated-content',
+    'page-autogenerated',
+    'page-generated-by',
+    'page-generated-from',
+    'page-generated-sources',
+    // Release that introduced the page's feature, and the product that
+    // release belongs to. Generated Connect reference pages set them on
+    // self-managed pages; the UI shows them as a "Since" badge.
+    'page-introduced-in',
+    'page-introduced-in-product',
     // Component-specific version attributes (from antora.yml)
     'latest-redpanda-tag', // Redpanda docker tag (e.g., v25.3.5)
     'latest-console-tag',
@@ -107,6 +210,29 @@ function generateFrontmatter(page) {
       return
     }
 
+    // Provenance travels as a JSON string in the attribute; the twin wants a list
+    if (key === 'page-generated-sources') {
+      try {
+        frontmatter[key] = JSON.parse(value)
+      } catch (err) {
+        frontmatter[key] = value
+      }
+      return
+    }
+    if (key === 'page-autogenerated') {
+      frontmatter[key] = value === '' || value === true || value === 'true'
+      return
+    }
+    if (key === 'page-introduced-in' || key === 'page-introduced-in-product') {
+      // Not boolean flags: an empty value (an unset attribute in a template)
+      // means there is nothing to record.
+      const text = String(value).trim()
+      if (!text) return
+      // A bare X.Y.Z, as the UI badge shows it.
+      frontmatter[key] = key === 'page-introduced-in' ? text.replace(/^v/i, '').replace(/\.+$/, '') : text
+      return
+    }
+
     // Skip empty attributes (AsciiDoc boolean flags)
     if (value === '') {
       // Special handling for version fields - use actual version from page source
@@ -294,8 +420,12 @@ function createEnterpriseFeatureRules () {
 }
 
 module.exports.createEnterpriseFeatureRules = createEnterpriseFeatureRules
+module.exports.generateFrontmatter = generateFrontmatter
 module.exports.formatStatusMarker = formatStatusMarker
 module.exports.ENTERPRISE_MARKER = ENTERPRISE_MARKER
+module.exports.readGeneratedMarker = readGeneratedMarker
+module.exports.findIncludeTargets = findIncludeTargets
+module.exports.collectGeneratedProvenance = collectGeneratedProvenance
 
 module.exports.register = function () {
   raiseListenerLimit(this)
@@ -556,6 +686,27 @@ module.exports.register = function () {
     return outerTurndown
   }
 
+  // Record which text on each page is generated, while the catalog still
+  // holds AsciiDoc: the page's own marker, and the marker on every include it
+  // resolves to. An extension that generates pages at build time sets
+  // page.generatedProvenance itself, and that wins.
+  this.on('contentClassified', ({ contentCatalog }) => {
+    const pages = contentCatalog.findBy({ family: 'page' })
+    let marked = 0
+    for (const page of pages) {
+      if (page.generatedProvenance) {
+        marked++
+        continue
+      }
+      const provenance = collectGeneratedProvenance(page, contentCatalog)
+      if (provenance.page || provenance.includes.length) {
+        page.generatedProvenance = provenance
+        marked++
+      }
+    }
+    logger.info(`Recorded generated-content provenance on ${marked} of ${pages.length} pages`)
+  })
+
   // Add marker attribute before UI rendering so templates can detect markdown availability
   this.on('documentsConverted', ({ contentCatalog }) => {
     const pages = contentCatalog.findBy({ family: 'page' })
@@ -568,6 +719,31 @@ module.exports.register = function () {
 
       // Add marker that UI templates can check
       page.asciidoc.attributes['page-has-markdown'] = ''
+
+      // Provenance as page attributes, so the twin's front matter and the UI
+      // templates can both read it. page-generated-content is set on every
+      // page, so "none" is a statement and not an absence.
+      const provenance = page.generatedProvenance
+      let generatedContent = 'none'
+      if (provenance?.page) {
+        generatedContent = 'all'
+        page.asciidoc.attributes['page-autogenerated'] = 'true'
+        const { generator, version, source } = provenance.page
+        if (generator) {
+          page.asciidoc.attributes['page-generated-by'] = `doc-tools${version ? ` ${version}` : ''} (${generator})`
+        }
+        // Only a named source: a legacy banner has none, and its prose is
+        // not one.
+        if (source) page.asciidoc.attributes['page-generated-from'] = source
+      } else if (provenance?.includes?.length) {
+        generatedContent = 'partial'
+      }
+      if (provenance?.includes?.length) {
+        page.asciidoc.attributes['page-generated-sources'] = JSON.stringify(
+          provenance.includes.map(({ include, generator, version, source }) => ({ include, generator, version, source }))
+        )
+      }
+      page.asciidoc.attributes['page-generated-content'] = generatedContent
     })
   })
 

package/extensions/generate-fields-only-pages.js

@@ -8,6 +8,7 @@ const os = require('os')
 
 
 const handlebars = require('handlebars')
+const { generatedMarker, DOC_TOOLS_VERSION } = require('../cli-utils/generated-marker')
 
 // Import and register Redpanda Connect helpers
 const helpers = require('../tools/redpanda-connect/helpers')
@@ -269,6 +270,19 @@ module.exports.register = function ({ config }) {
           // Mark as field-only page (used by convert-to-markdown and add-llms-directive to skip directive)
           file.isFieldOnlyPage = true
 
+          // The whole page is generated here at build time, so say so the way
+          // a file marker would; convert-to-markdown exports it as
+          // page-autogenerated and page-generated-from in the twin.
+          file.generatedProvenance = {
+            page: {
+              generator: 'fields-only-pages',
+              version: DOC_TOOLS_VERSION,
+              source: 'the component spec in redpanda-data/connect',
+              note: generatedMarker('fields-only-pages', 'the component spec in redpanda-data/connect').replace(/^\/\/\s*/, ''),
+            },
+            includes: [],
+          }
+
           pagesGenerated++
         } catch (err) {
           logger.error(`Failed to generate field-only page for ${type}/${item.name}: ${err.message}`)

package/extensions/modify-connect-tag-playbook.js

@@ -526,7 +526,7 @@ module.exports.register = function ({ config }) {
     if (r.kept + r.outsideGenerated + r.providedElsewhere + r.otherComponents === 0) return
     logger.info(
       `Redpanda Connect content: kept ${r.kept} generated files; skipped ${r.providedElsewhere} already provided by another source, ` +
-      `${r.outsideGenerated} outside the generated partials, examples, and attachments, and ${r.otherComponents} from other components`
+      `${r.outsideGenerated} outside the generated partials and examples, and ${r.otherComponents} from other components`
     )
   })
 
@@ -592,7 +592,7 @@ module.exports.register = function ({ config }) {
       from = `the ${tag} ${asset.ASSET_NAME} release asset`
     }
     if (!files.some((f) => asset.toResource(f.path))) {
-      throw new Error(`${from} has no files under modules/components/partials or modules/components/examples`)
+      throw new Error(`${from} has no files under modules/components/partials, modules/components/examples, or modules/components/attachments`)
     }
     const r = addConnectDocs(contentCatalog, files, origin, { replace: !!localDir })
     const versions = r.versions.filter(Boolean)