@redpanda-data/docs-extensions-and-macros 4.2.5 → 4.4.0

package/README.adoc

@@ -123,7 +123,7 @@ antora:
         - home:ROOT:attachment$custom-file.txt
 ----
 
-==== Registration example
+==== Registration
 
 [source,yaml]
 ----
@@ -155,7 +155,7 @@ Any elements, classes, or IDs that you want to exclude from the index.
 index-latest-only (optional)::
 Whether to index all versions or just the latest version of a component.
 
-==== Registration example
+==== Registration
 
 ```yaml
 antora:
@@ -246,7 +246,7 @@ This extension does not require any environment variables.
 
 There are no configurable options for this extension.
 
-==== Registration example
+==== Registration
 
 ```yaml
 antora:
@@ -297,7 +297,7 @@ antora:
             upgrade_doc: ROOT:upgrade:index.adoc
 ----
 
-==== Registration example
+==== Registration
 
 You can register the extension with a customized configuration for different components in your playbook:
 
@@ -456,7 +456,7 @@ NOTE: If you don't set the environment variable, the latest version of Redpanda
 
 There are no configurable options for this extension.
 
-==== Registration Example
+==== Registration
 
 ```yaml
 antora:
@@ -488,7 +488,7 @@ The following attributes are available to the latest version of the `ROOT` compo
 
 NOTE: If you don't set the environment variable, the latest versions may not be fetched. When the environment variable is not set, the extension sends unauthenticated requests to GitHub. Unauthenticated requests may result in hitting the API rate limit and cause GitHub to reject the request.
 
-==== Registration example
+==== Registration
 
 ```yaml
 antora:
@@ -508,7 +508,7 @@ This extension does not require any environment variables.
 
 There are no configurable options for this extension. It operates based on site attributes defined in `add-global-attributes.js` to determine valid categories and subcategories.
 
-==== Registration example
+==== Registration
 
 Register the `validate-attributes` extension in the Antora playbook under the `antora.extensions` key like so:
 
@@ -531,7 +531,7 @@ This extension operates without requiring any specific environment variables.
 
 This extension does not offer configurable options. It uses the inherent attributes of pages to determine relationships based on `page-categories` and deployment types (`env-kubernetes`, `env-linux`, `env-docker`, `page-cloud`).
 
-==== Registration example
+==== Registration
 
 To integrate the `related-docs-extension` into your Antora playbook, add it under the `antora.extensions` key as demonstrated below:
 
@@ -554,7 +554,7 @@ This extension does not require any environment variables.
 
 The extension operates without explicit configuration options. It automatically processes documentation pages to identify and link related labs based on shared `page-categories` attributes and deployment types (`env-kubernetes`, `env-linux`, `env-docker`, `page-cloud`).
 
-==== Registration example
+==== Registration
 
 Include the `related-labs-extension` in the Antora playbook under the `antora.extensions` key as follows:
 
@@ -579,7 +579,7 @@ The extension accepts the following configuration options:
 
 attributespath (optional):: Specifies the path to a local YAML file that contains global attributes. If this is provided, the extension will load attributes from this file first. If this path is not provided or no valid attributes are found in the file, the extension will fall back to loading attributes from the `shared` component.
 
-==== Registration example
+==== Registration
 
 ```yml
 antora:
@@ -602,7 +602,7 @@ This extension does not require any environment variables.
 
 There are no configurable options for this extension.
 
-==== Registration example
+==== Registration
 
 ```yaml
 antora:
@@ -640,7 +640,7 @@ data.replacements (required):: An array of replacement configurations. Each conf
 
 NOTE: Ensure that `file_patterns` accurately reflect the paths of the attachments you want to process. Overly broad patterns may include unintended files, while overly restrictive patterns might exclude necessary resources.
 
-==== Registration Example
+==== Registration
 
 This is an example of how to register and configure the `replace-attributes-in-attachments` extension in your Antora playbook. This example demonstrates defining multiple replacement configurations, each targeting different components and specifying their own file patterns and custom replacements.
 
@@ -698,7 +698,7 @@ Term files should follow the following structure:
 This is the detailed description of the term.
 ```
 
-==== Registration example
+==== Registration
 
 ```yml
 antora:
@@ -729,7 +729,7 @@ Whether to add unlisted pages to the navigation. The default is `false` (unliste
 unlistedPagesHeading (optional)::
 The heading under which to list the unlisted pages in the navigation. The default is 'Unlisted Pages'.
 
-==== Registration example
+==== Registration
 
 ```yaml
 antora:
@@ -749,7 +749,7 @@ IMPORTANT: Be sure to register each extension under the `asciidoc.extensions` ke
 
 This extension adds the necessary classes to make line numbers and line highlighting work with Prism.js.
 
-==== Registration example
+==== Registration
 
 ```yaml
 antora:
@@ -763,6 +763,108 @@ This section documents the Asciidoc macros that are provided by this library and
 
 IMPORTANT: Be sure to register each extension under the `asciidoc.extensions` key in the playbook, not the `antora.extensions` key.
 
+=== data_template
+
+The `data_template` block processor lets you render dynamic AsciiDoc content from external or local data sources (JSON, YAML, or plain text) using Handlebars templates.
+
+This is useful for generating documentation from structured data like config fields, component metadata, or examples.
+
+=== Usage
+
+You can use the `data_template` block macro to dynamically generate AsciiDoc content from structured data files such as JSON or YAML. The macro uses a Handlebars template to render the data.
+
+[source,asciidoc]
+----
+[data_template, ROOT:example$connect.json]
+--
+== Component: {{{name}}}
+
+Summary: {{{summary}}}
+
+{{#each fields}}
+=== {{name}}
+
+*Type*: `{{type}}`
+
+{{{description}}}
+{{/each}}
+--
+----
+
+The block content is a Handlebars template. Fields from the data file are injected into this template during site build.
+
+The macro accepts one or two positional attributes:
+
+1. The first attribute (`dataPath`) is required and should be the resource ID of the data file.
+2. The second attribute (`overrides`) is optional and allows you to override or merge values from a secondary file.
+
+You can apply overrides by specifying a second file:
+
+[source,asciidoc]
+----
+[data_template, ROOT:example$connect.json, ROOT:example$overrides.json]
+--
+...template content...
+--
+----
+
+In this case:
+
+- The macro first loads and parses `connect.json`.
+- Then it loads `overrides.json` and **merges** its values into the base file.
+- Arrays of objects (such as fields or processors) are merged by matching objects with the same `name` key.
+- The result is passed into the Handlebars template.
+
+This is useful for tweaking content (like updating a `description`) without modifying the original source file.
+
+==== Triple mustaches
+
+When your data contains AsciiDoc markup (like lists, admonitions, or headings), use triple curly braces:
+
+[source,handlebars]
+----
+{{{description}}}
+----
+
+This tells Handlebars to not escape the content, so Asciidoctor can render it correctly.
+
+==== Handlebars helpers
+
+The following helpers are available inside templates:
+
+* `eq`, `ne` — Equality helpers.
+* `uppercase` — Converts text to uppercase.
+* `renderConnectFields` — Renders config fields for Redpanda Connect.
+* `renderConnectExamples` — Renders usage examples.
+* `selectByJsonPath` — Selects items from the data using a JSONPath expression.
+
+==== Registration
+
+To use this macro, register it in your Antora playbook:
+
+[source,yaml]
+----
+asciidoc:
+  extensions:
+    - '@redpanda-data/docs-extensions-and-macros/macros/data-template'
+----
+
+==== Example
+
+You can use the `selectByJsonPath` helper to filter data. For example, if you want to render only the `redis` processor's fields from a JSON file, you can do it like this:
+
+[source,asciidoc]
+----
+[data_template, redpanda-connect:ROOT:example$connect.json]
+--
+{{#selectByJsonPath this "$.processors[?(@.name=='redis')]" }}
+{{renderConnectFields this.config.children}}
+{{/selectByJsonPath}}
+--
+----
+
+This will render only the fields for the `redis` processor.
+
 === config_ref
 
 This inline macro is used to generate a reference to a configuration value in the Redpanda documentation. The macro's parameters allow for control over the generated reference's format and the type of output produced.
@@ -798,7 +900,7 @@ For example:
 config_ref:example_config,true,tunable-properties[]
 ----
 
-==== Registration example
+==== Registration
 
 [,yaml]
 ----
@@ -807,6 +909,67 @@ asciidoc:
     - '@redpanda-data/docs-extensions-and-macros/macros/config-ref'
 ----
 
+=== data_template
+
+The `data_template` macro provides a way to dynamically generate AsciiDoc content by combining external or local data sources with Handlebars templates. When you use the `data_template` block macro, the extension performs the following steps:
+
+* Resolves the `dataPath` attribute to locate a data file (in JSON, YAML, or raw text format).
+* Fetches and caches external resources or reads local files from the Antora content catalog.
+* Parses the data file.
+* Compiles the block's content as a Handlebars template, injecting the parsed data.
+* Processes the resulting text as AsciiDoc using Asciidoctor to generate the final HTML output.
+
+By default, Handlebars escapes HTML to prevent potential security issues. However, if your data includes AsciiDoc markup (such as headings, lists, or formatting directives), escaping it will prevent Asciidoctor from converting the markup correctly.
+
+To ensure that your AsciiDoc syntax is preserved during template rendering, **use the triple curly braces syntax** in your Handlebars templates. For example, if your JSON file contains a `description` field with AsciiDoc content, reference it like this:
+
+[source,handlebars]
+----
+{{{description}}}
+----
+
+This tells Handlebars to output the content unescaped, allowing Asciidoctor to process the raw AsciiDoc markup correctly.
+
+==== Usage
+
+In an AsciiDoc document, you can invoke the data template macro as follows:
+
+[,asciidoc]
+----
+[data_template, ROOT:example$connect.json]
+--
+Version: {{{version}}}
+
+{{#each buffers}}
+
+=== {{{this.name}}}
+
+Status: {{{this.status}}}
+
+{{#if (eq this.name 'memory')}}
+This is a custom description for the memory buffer.
+{{else}}
+{{{this.summary}}}
+{{/if}}
+
+{{/each}}
+
+--
+----
+
+==== Registration
+
+Register the macro in your Antora playbook under the `asciidoc.extensions` key:
+
+[source,yaml]
+----
+asciidoc:
+  extensions:
+    - require: '@redpanda-data/docs-extensions-and-macros/macros/data-template'
+----
+
+This configuration ensures that during the build process, the data template macro is executed to fetch, parse, and render data as part of your docs.
+
 === glossterm
 
 The `glossterm` inline macro provides a way to define and reference glossary terms in your AsciiDoc documents.
@@ -855,7 +1018,7 @@ Whether to enable tooltips for the defined terms. Valid values are:
 
 The last two options are intended to support js/css tooltip solutions such as tippy.js.
 
-==== Registration example
+==== Registration
 
 [,yaml]
 ----
@@ -897,7 +1060,7 @@ For default values and documentation for configuration options, see the https://
 
 If you do not specify a Helm reference value, the macro generates a link without specifying a path.
 
-==== Registration example
+==== Registration
 
 [,yaml]
 ----
@@ -918,7 +1081,7 @@ The categories are fetched from the `connectCategoriesData` that's generated in
 components_by_category::[<type>]
 ```
 
-==== Registration example
+==== Registration
 
 ```yaml
 asciidoc:
@@ -938,7 +1101,7 @@ The types are fetched from the `flatComponentsData` that's generated in the <<Co
 component_table::[]
 ```
 
-==== Registration example
+==== Registration
 
 ```yaml
 asciidoc:
@@ -958,7 +1121,7 @@ The types are fetched from the `flatComponentsData` that's generated in the <<Co
 component_type_dropdown::[]
 ```
 
-==== Registration example
+==== Registration
 
 ```yaml
 asciidoc:

package/bin/doc-tools.js

@@ -0,0 +1,328 @@
+#!/usr/bin/env node
+
+const { execSync, spawnSync } = require('child_process');
+const { Command } = require('commander');
+const path = require('path');
+const fs = require('fs');
+
+// --------------------------------------------------------------------
+// Dependency check functions
+// --------------------------------------------------------------------
+function checkDependency(command, versionArg, name, helpURL) {
+  try {
+    execSync(`${command} ${versionArg}`, { stdio: 'ignore' });
+  } catch (error) {
+    console.error(`Error: ${name} is required but not found or not working properly.
+Please install ${name} and try again.
+For more info, see: ${helpURL}`);
+    process.exit(1);
+  }
+}
+
+function checkCommandExists(command) {
+  try {
+    execSync(`which ${command}`, { stdio: 'ignore' });
+    return true;
+  } catch (error) {
+    console.error(`Error: \`${command}\` is required but not found. Please install \`${command}\` and try again.`);
+    return false;
+  }
+}
+
+function checkMake() {
+  if (!checkCommandExists('make')) {
+    console.error('Error: `make` is required but not found. Please install `make` to use the automation Makefile. For help, see: https://www.google.com/search?q=how+to+install+make');
+    process.exit(1);
+  }
+}
+
+function checkPython() {
+  const candidates = ['python3', 'python'];
+  let found = false;
+
+  for (const cmd of candidates) {
+    try {
+      const versionOutput = execSync(`${cmd} --version`, {
+        encoding: 'utf8',
+        stdio: ['pipe', 'pipe', 'ignore']
+      }).trim();
+      // versionOutput looks like "Python 3.x.y"
+      const versionString = versionOutput.split(' ')[1];
+      const [major, minor] = versionString.split('.').map(Number);
+      if (major > 3 || (major === 3 && minor >= 10)) {
+        found = true;
+        break;
+      } else {
+        console.error(`Error: Python 3.10 or higher is required. Detected version: ${versionString}`);
+        process.exit(1);
+      }
+    } catch {
+      // this candidate didn’t exist or errored—try the next one
+    }
+  }
+  if (!found) {
+    console.error('Error: Python 3.10 or higher is required but not found.\nPlease install Python and ensure `python3 --version` or `python --version` returns at least 3.10: https://www.geeksforgeeks.org/how-to-install-python-on-mac/');
+    process.exit(1);
+  }
+}
+
+function checkCompiler() {
+  const gccInstalled = checkCommandExists('gcc');
+  const clangInstalled = checkCommandExists('clang');
+  if (!gccInstalled && !clangInstalled) {
+    console.error('Error: A C++ compiler (such as gcc or clang) is required but not found. Please install one: https://osxdaily.com/2023/05/02/how-install-gcc-mac/');
+    process.exit(1);
+  }
+}
+
+function checkDocker() {
+  checkDependency('docker', '--version', 'Docker', 'https://docs.docker.com/get-docker/');
+  try {
+    execSync('docker info', { stdio: 'ignore' });
+  } catch (error) {
+    console.error('Error: Docker daemon appears to be not running. Please start Docker.');
+    process.exit(1);
+  }
+}
+
+function verifyPropertyDependencies() {
+  checkMake();
+  checkPython();
+  checkCompiler();
+}
+
+function verifyMetricsDependencies() {
+  checkPython();
+  if (!checkCommandExists('curl') || !checkCommandExists('tar')) {
+    // `checkCommandExists` already prints a helpful message.
+    process.exit(1);
+  }
+  checkDocker();
+}
+// --------------------------------------------------------------------
+// Main CLI Definition
+// --------------------------------------------------------------------
+const programCli = new Command();
+
+programCli
+  .name('doc-tools')
+  .description('Redpanda Document Automation CLI')
+  .version('1.0.0');
+
+// Top-level commands.
+programCli
+  .command('install-test-dependencies')
+  .description('Install packages for doc test workflows')
+  .action(() => {
+    const scriptPath = path.join(__dirname, '../cli-utils/install-test-dependencies.sh');
+    const result = spawnSync(scriptPath, { stdio: 'inherit', shell: true });
+    process.exit(result.status);
+  });
+
+programCli
+  .command('get-redpanda-version')
+  .description('Print the latest Redpanda version')
+  .option('--beta', 'Return the latest RC (beta) version if available')
+  .option('--from-antora', 'Read prerelease flag from local antora.yml')
+  .action(async (options) => {
+    try {
+      await require('../tools/get-redpanda-version.js')(options);
+    } catch (err) {
+      console.error(err);
+      process.exit(1);
+    }
+  });
+
+programCli
+  .command('get-console-version')
+  .description('Print the latest Console version')
+  .option('--beta', 'Return the latest beta version if available')
+  .option('--from-antora', 'Read prerelease flag from local antora.yml')
+  .action(async (options) => {
+    try {
+      await require('../tools/get-console-version.js')(options);
+    } catch (err) {
+      console.error(err);
+      process.exit(1);
+    }
+  });
+
+// Create an "automation" subcommand group.
+const automation = new Command('generate')
+  .description('Run docs automations (properties, metrics, and rpk docs generation)');
+
+// --------------------------------------------------------------------
+// Automation Subcommands: Delegate to a unified Bash script internally.
+// --------------------------------------------------------------------
+
+// Common options for both automation tasks.
+const commonOptions = {
+  tag: 'latest',
+  dockerRepo: 'redpanda',
+  consoleTag: 'latest',
+  consoleDockerRepo: 'console'
+};
+
+function runClusterDocs(mode, tag, options) {
+  const script = path.join(__dirname, '../cli-utils/generate-cluster-docs.sh');
+  const args   = [ mode, tag, options.dockerRepo, options.consoleTag, options.consoleDockerRepo ];
+  console.log(`Running ${script} with arguments: ${args.join(' ')}`);
+  const r = spawnSync('bash', [ script, ...args ], { stdio: 'inherit', shell: true });
+  if (r.status !== 0) process.exit(r.status);
+}
+
+// helper to diff two autogenerated directories
+function diffDirs(kind, oldTag, newTag) {
+  const oldDir  = path.join('autogenerated', oldTag, kind);
+  const newDir  = path.join('autogenerated', newTag, kind);
+  const diffDir = path.join('autogenerated', 'diffs', kind, `${oldTag}_to_${newTag}`);
+  const patch   = path.join(diffDir, 'changes.patch');
+
+  if (!fs.existsSync(oldDir)) {
+    console.error(`❌ Cannot diff: missing ${oldDir}`);
+    process.exit(1);
+  }
+  if (!fs.existsSync(newDir)) {
+    console.error(`❌ Cannot diff: missing ${newDir}`);
+    process.exit(1);
+  }
+
+  fs.mkdirSync(diffDir, { recursive: true });
+
+  const cmd = `diff -ru "${oldDir}" "${newDir}" > "${patch}" || true`;
+  const res = spawnSync(cmd, { stdio: 'inherit', shell: true });
+
+  if (res.error) {
+    console.error(`❌ diff failed: ${res.error.message}`);
+    process.exit(1);
+  }
+  console.log(`✅ Wrote patch: ${patch}`);
+}
+
+automation
+  .command('metrics-docs')
+  .description('Extract Redpanda metrics and generate JSON/AsciiDoc docs')
+  .option('--tag <tag>', 'Redpanda tag (default: latest)', commonOptions.tag)
+  .option('--docker-repo <repo>', '...', commonOptions.dockerRepo)
+  .option('--console-tag <tag>', '...', commonOptions.consoleTag)
+  .option('--console-docker-repo <repo>', '...', commonOptions.consoleDockerRepo)
+  .option('--diff <oldTag>', 'Also diff autogenerated metrics from <oldTag> → <tag>')
+  .action((options) => {
+    verifyMetricsDependencies();
+
+    const newTag = options.tag;
+    const oldTag = options.diff;
+
+    if (oldTag) {
+      const oldDir = path.join('autogenerated', oldTag, 'metrics');
+      if (!fs.existsSync(oldDir)) {
+        console.log(`⏳ Generating metrics docs for old tag ${oldTag}…`);
+        runClusterDocs('metrics', oldTag, options);
+      }
+    }
+
+    console.log(`⏳ Generating metrics docs for new tag ${newTag}…`);
+    runClusterDocs('metrics', newTag, options);
+
+    if (oldTag) {
+      diffDirs('metrics', oldTag, newTag);
+    }
+
+    process.exit(0);
+  });
+
+automation
+  .command('property-docs')
+  .description('Extract properties from Redpanda source')
+  .option('--tag <tag>', 'Git tag or branch to extract from (default: dev)', 'dev')
+  .option('--diff <oldTag>', 'Also diff autogenerated properties from <oldTag> → <tag>')
+  .action((options) => {
+    verifyPropertyDependencies();
+
+    const newTag = options.tag;
+    const oldTag = options.diff;
+    const cwd    = path.resolve(__dirname, '../tools/property-extractor');
+    const make   = (tag) => {
+      console.log(`⏳ Building property docs for ${tag}…`);
+      const r = spawnSync('make', ['build', `TAG=${tag}`], { cwd, stdio: 'inherit' });
+      if (r.error  ) { console.error(r.error); process.exit(1); }
+      if (r.status !== 0) process.exit(r.status);
+    };
+
+    if (oldTag) {
+      const oldDir = path.join('autogenerated', oldTag, 'properties');
+      if (!fs.existsSync(oldDir)) make(oldTag);
+    }
+
+    make(newTag);
+
+    if (oldTag) {
+      diffDirs('properties', oldTag, newTag);
+    }
+
+    process.exit(0);
+  });
+
+automation
+  .command('rpk-docs')
+  .description('Generate documentation for rpk commands')
+  .option('--tag <tag>', 'Redpanda tag (default: latest)', commonOptions.tag)
+  .option('--docker-repo <repo>', '...', commonOptions.dockerRepo)
+  .option('--console-tag <tag>', '...', commonOptions.consoleTag)
+  .option('--console-docker-repo <repo>', '...', commonOptions.consoleDockerRepo)
+  .option('--diff <oldTag>', 'Also diff autogenerated rpk docs from <oldTag> → <tag>')
+  .action((options) => {
+    verifyMetricsDependencies();
+
+    const newTag = options.tag;
+    const oldTag = options.diff;
+
+    if (oldTag) {
+      const oldDir = path.join('autogenerated', oldTag, 'rpk');
+      if (!fs.existsSync(oldDir)) {
+        console.log(`⏳ Generating rpk docs for old tag ${oldTag}…`);
+        runClusterDocs('rpk', oldTag, options);
+      }
+    }
+
+    console.log(`⏳ Generating rpk docs for new tag ${newTag}…`);
+    runClusterDocs('rpk', newTag, options);
+
+    if (oldTag) {
+      diffDirs('rpk', oldTag, newTag);
+    }
+
+    process.exit(0);
+  });
+
+programCli
+.command('fetch')
+.description('Fetch a file or directory from GitHub and save locally')
+.requiredOption('-o, --owner <owner>', 'GitHub repo owner or org')
+.requiredOption('-r, --repo <repo>', 'GitHub repo name')
+.requiredOption('-p, --remote-path <path>', 'Path in the repo to fetch')
+.requiredOption('-d, --save-dir <dir>', 'Local directory to save into')
+.option('-f, --filename <name>', 'Custom filename to save as')
+.action(async (options) => {
+  try {
+    const fetchFromGithub = await require('../tools/fetch-from-github.js');
+    // options.owner, options.repo, options.remotePath, options.saveDir, options.filename
+    await fetchFromGithub(
+      options.owner,
+      options.repo,
+      options.remotePath,
+      options.saveDir,
+      options.filename
+    );
+  } catch (err) {
+    console.error('❌', err.message);
+    process.exit(1);
+  }
+});
+
+
+// Attach the automation group to the main program.
+programCli.addCommand(automation);
+
+programCli.parse(process.argv);
+