@redpanda-data/docs-extensions-and-macros 2.5.0

package/README.adoc

@@ -0,0 +1,422 @@
+= Antora Extensions and Macros for Redpanda Docs
+:url-project: https://github.com/JakeSCahill/antora-extensions-and-macros
+:url-git: https://git-scm.com
+:url-git-dl: {url-git}/downloads
+:url-nodejs: https://nodejs.org
+:url-nodejs-releases: https://github.com/nodejs/Release#release-schedule
+:url-nvm-install: {url-nvm}#installation
+:idprefix:
+:idseparator: -
+ifdef::env-github[]
+:important-caption: :exclamation:
+:note-caption: :paperclip:
+endif::[]
+:toc:
+:toc-title: Contents
+
+toc::[]
+
+This library provides https://docs.antora.org/antora/latest/extend/extensions/[Antora extensions] and https://docs.asciidoctor.org/asciidoctor.js/latest/extend/extensions/register/[Asciidoc macros] developed for Redpanda documentation.
+
+== Prerequisites
+
+To use this library, you must have {url-nodejs}[Node.js] 16 or higher installed on your machine.
+
+[,bash]
+----
+node --version
+----
+
+If this command fails with an error, you don't have Node.js installed.
+
+When you have Node.js installed, use the following command to install the `antora-extensions-and-macros` package in your project:
+
+[,bash]
+----
+npm i antora-extensions-and-macros
+----
+
+To use the development version instead, refer to the <<development-quickstart,Development Quickstart>>.
+
+== Extensions
+
+This section documents the Antora extensions that are provided by this library and how to configure them.
+
+IMPORTANT: Be sure to register each extension under the `antora.extensions` key in the playbook, not the `asciidoc.extensions` key.
+
+=== Algolia indexer
+
+This extension generates an Algolia index for each version of each component. The index entries are then saved to Algolia using the `saveObjects()` method, and also saved as JSON files in the site catalog. JSON files are published to the site root using the following template: `algolia-<component>-<version>.json`.
+
+NOTE: Only pages that include an `<article>` element with the `doc` class are indexed. Pages marked as "noindex" for "robots" are skipped.
+
+==== Environment variables
+
+- `ALGOLIA_ADMIN_API_KEY` (required)
+- `ALGOLIA_APP_ID` (required)
+- `ALGOLIA_INDEX_NAME` (required)
+
+==== Configuration options
+
+The extension accepts the following configuration options:
+
+excludes (optional)::
+Any elements, classes, or IDs that you want to exclude from the index.
+index-latest-only (optional)::
+Whether to index all versions or just the latest version of a component.
+
+==== Registration example
+
+```yaml
+antora:
+  extensions:
+  - require: 'node_modules/antora-extensions-and-macros/extensions/algolia-indexer/index.js'
+    excludes: ['.thumbs','script', '.page-versions','.feedback-section','.banner-container']
+    index-latest-only: true
+```
+
+=== Version fetcher
+
+This extension fetches the latest release tag and latest release commit hash from http://github.com/redpanda-data/redpanda. These values are then assigned to the `full-version` and `latest-release-commit` attributes of the latest version of the Redpanda documentation, respectively.
+
+It also fetches the latest version of Redpanda Console and assigns it to the `latest-console-version` attribute in the playbook so that all components have access to it.
+
+==== Environment variables
+
+- `REDPANDA_GITHUB_TOKEN` (optional): A Personal access token (PAT) that has `repo` permissions for the `redpanda-data` GitHub organization.
+
+NOTE: If you don't set the environment variable, the latest versions may not be made available to your build. When the environment variable is not set, the extension sends unauthenticated requests to GitHub. Unauthenticated requests may result in hitting the API rate limit and cause GitHub to reject the request.
+
+==== Registration example
+
+```yaml
+antora:
+  extensions:
+  - 'node_modules/antora-extensions-and-macros/extensions/version-fetcher/set-latest-version.js'
+```
+
+=== Global attributes
+
+This extension fetches the content of all YAML files in a GitHub directory and merges the contents with the `asciidoc.attributes` object in the Antora playbook. This allows you to define Asciidoc attributes in an external repository and automatically include them in your documentation.
+
+[IMPORTANT]
+====
+- The GitHub directory that contains the global attributes must be named `global-attributes`.
+- Only YAML files are supported. Other types of files are ignored.
+- If a key is present in both the global attributes and the playbook's `asciidoc.attributes`, the value in the playbook takes precedence.
+====
+
+==== Environment variables
+
+- `GLOBAL_ATTRIBUTES_GITHUB_TOKEN` (optional): A Personal access token (PAT) that has `repo` permissions for the GitHub organization defined in `org`.
+
+NOTE: If you don't set the environment variable, global attributes may not be made available to your build. When the environment variable is not set, the extension sends unauthenticated requests to GitHub. Unauthenticated requests may result in hitting the API rate limit and cause GitHub to reject the request.
+
+==== Configuration options
+
+The extension accepts the following configuration options:
+
+org (required)::
+The GitHub organization that owns the repository.
+
+repo (required)::
+The name of the repository.
+
+branch (required)::
+The branch in the repository where the global attributes are located.
+
+==== Registration example
+
+```yaml
+antora:
+  extensions:
+  - require: 'node_modules/antora-extensions-and-macros/extensions/add-global-attributes.js'
+    org: example
+    repo: test
+    branch: main
+```
+
+=== Replace attributes in attachments
+
+This extension replaces AsciiDoc attribute placeholders with their respective values in attachment files, such as CSS, HTML, and YAML.
+
+[IMPORTANT]
+====
+- By default, this extension processes attachments for the `ROOT` (redpanda) component only. This behavior is hardcoded and cannot be changed in the configuration.
+- The `@` character is removed from attribute values to prevent potential issues with CSS or HTML syntax.
+- If the same attribute placeholder is used multiple times within a file, all instances will be replaced with the attribute's value.
+====
+
+==== Registration example
+
+```yaml
+antora:
+  extensions:
+  - 'node_modules/antora-extensions-and-macros/extensions/replace-attributes-in-attachments.js'
+```
+
+=== Unlisted pages
+
+This extension identifies and logs any pages that aren't listed in the navigation (nav) file of each version of each component. It then optionally adds these unlisted pages to the end of the navigation tree under a configurable heading.
+
+IMPORTANT: By default, this extension excludes components named 'api'. This behavior is hardcoded and cannot be changed in the configuration.
+
+==== Configuration options
+
+This extension accepts the following configuration options:
+
+addToNavigation (optional)::
+Whether to add unlisted pages to the navigation. The default is `false` (unlisted pages are not added).
+
+unlistedPagesHeading (optional)::
+The heading under which to list the unlisted pages in the navigation. The default is 'Unlisted Pages'.
+
+==== Registration example
+
+```yaml
+antora:
+  extensions:
+  - require: 'node_modules/antora-extensions-and-macros/extensions/unlisted-pages.js'
+    addToNavigation: true
+    unlistedPagesHeading: 'Additional Resources'
+```
+
+== Macros
+
+This section documents the Asciidoc macros that are provided by this library and how to configure them.
+
+IMPORTANT: Be sure to register each extension under the `asciidoc.extensions` key in the playbook, not the `antora.extensions` key.
+
+=== config_ref
+
+This inline macro is used to generate a reference to a configuration value in the Redpanda documentation. The macro's parameters allow for control over the generated reference's format and the type of output produced.
+
+==== Usage
+
+The `config_ref` macro is used in an AsciiDoc document as follows:
+
+[,asciidoc]
+----
+config_ref:configRef,isLink,path[]
+----
+
+The `config_ref` macro takes three parameters:
+
+configRef::
+This is the configuration reference, which is also used to generate the anchor link if `isLink` is `true`.
+
+isLink::
+Whether the output should be a link. If `isLink` is set to `true`, the output will be a cross-reference (xref) to the relevant configuration value.
+
+path::
+This is the path to the document where the configuration value is defined. This parameter is used to to generate the link if `isLink` is `true`.
+
+IMPORTANT: The path must be the name of a document at the root of the `reference` module.
+
+NOTE: The `config_ref` macro is environment-aware. It checks if the document it is being used in is part of a Kubernetes environment by checking if the `env-kubernetes` attribute is set in the document's attributes. Depending on this check, it either prepends `storage.tieredConfig.` to the `configRef` or just uses the `configRef` as is.
+
+For example:
+
+[,asciidoc]
+----
+config_ref:example_config,true,tunable-properties[]
+----
+
+==== Registration example
+
+[,yaml]
+----
+asciidoc:
+  extensions:
+    - 'node_modules/antora-extensions-and-macros/macros/config-ref.js'
+----
+
+=== glossary and glossterm
+
+The glossary module provides a way to define and reference glossary terms in your AsciiDoc documents.
+
+This module consists of two parts: a block macro (`glossary`) and an inline macro (`glossterm`).
+
+NOTE: This macro is a customized version of https://gitlab.com/djencks/asciidoctor-glossary[`asciidoctor-glossary`]. We added the ability to define terms globally in <<global-attributes, global attributes>> as well as link to external URLs.
+
+==== Usage
+
+To insert a glossary dlist, use the glossary block macro.
+
+[,asciidoc]
+----
+glossary::[]
+----
+
+Glossary terms defined in the `glossterm` inline macro before the `glossary` macro is used appear as a definition list, sorted by term.
+
+The `glossterm` inline macro is used to reference a term within the text of the document:
+
+[,asciidoc]
+----
+glossterm:myTerm[myDefinition]
+----
+
+It takes two parameters:
+
+term::
+The term to be defined.
+
+definition (optional)::
+The definition of the term. If the term is defined in the <<global-attributes, global attributes>>, you can omit the definition as it will always be replaced by the definition in the global attributes.
+
+==== Configuration options
+
+glossary-log-terms (optional)::
+Whether to log a textual representation of a definition list item to the console.
+
+glossary-term-role (optional)::
+Role to assign each term. By default, glossary terms are assigned the `glossary-term` role, which gives them the class `glossary-term` in generated html.
+
+glossary-links (optional)::
+Whether to generate links to glossary entries.
+By default, links to the glossary entries are generated from the glossary terms. To avoid this, set the attribute to `false` as either asciidoctor configuration or a header attribute.
+
+glossary-page (optional)::
+Target page for glossary links. By default, links are generated to the same page as the glossary term. To specify the target page, set this attribute to the resource ID of a page where the `glossary` block macro is used.
+
+glossary-tooltip (optional)::
+Whether to enable tooltips for the defined terms. Valid values are:
+- title: This uses the browser built-in `title` attribute to display the definition.
+
+- true: This inserts the definition as the value of the attribute `data-glossary-tooltip`.
+
+- data-<attribute-name>​: This inserts the definition as the value of the supplied attribute name, which must start with `data`.
+
+The last two options are intended to support js/css tooltip solutions such as tippy.js.
+
+[IMPORTANT]
+.Multi-page use
+====
+In Antora, a glossary is constructed for each component-version.
+When the `glossary` block macro is evaluated, only terms known as of the rendering can be included.
+Therefore, it is necessary that the page containing this macro in a component-version be rendered last.
+It may be possible to arrange this by naming the page starting with a lot of 'z’s, such as `zzzzzz-glossary.adoc`.
+====
+
+==== Registration example
+
+[,yaml]
+----
+asciidoc:
+  extensions:
+    - 'node_modules/antora-extensions-and-macros/macros/glossary.js'
+----
+
+=== helm_ref
+
+This is an inline macro to create links to a Helm `values.yaml` file on ArtifactHub.
+
+==== Usage
+
+In an AsciiDoc document, use the `helm_ref` macro as follows:
+
+[,asciidoc]
+----
+helm_ref:<helmRef>[]
+----
+
+Where `<helmRef>` is the Helm configuration value you want to reference in the `values.yaml` file.
+
+For example:
+
+Given a Helm reference value of `myConfigValue`, you would use the macro like this:
+
+[,asciidoc]
+----
+helm_ref:myConfigValue[]
+----
+
+This will generate the following output:
+
+[,asciidoc]
+----
+For default values and documentation for configuration options, see the https://artifacthub.io/packages/helm/redpanda-data/redpanda?modal=values&path=myConfigValue[values.yaml] file.
+----
+
+If you do not specify a Helm reference value, the macro generates a link without specifying a path.
+
+==== Registration example
+
+[,yaml]
+----
+asciidoc:
+  extensions:
+    - 'node_modules/antora-extensions-and-macros/macros/helm-ref.js'
+----
+
+== Development quickstart
+
+This section provides information on how to develop this project.
+
+=== Prerequisites
+
+To build this project, you need the following software installed on your computer:
+
+* {url-git}[git] (command: `git`)
+* {url-nodejs}[Node.js] (commands: `node`, `npm`, and `npx`)
+
+==== git
+
+Make sure you have git installed.
+
+[,bash]
+----
+git --version
+----
+
+If not, {url-git-dl}[download and install] the git package for your system.
+
+==== Node.js
+
+Make sure that you have Node.js installed (which also provides npm and npx).
+
+[,bash]
+----
+node --version
+----
+
+If this command fails with an error, you don't have Node.js installed.
+
+Now that you have git and Node.js installed, you're ready to start developing on this project.
+
+=== Clone the project
+
+Clone the project using git:
+
+[,bash,subs=attributes+]
+----
+git clone {url-project}
+----
+
+Change into the project directory and stay in this directory when running all subsequent commands.
+
+=== Install dependencies
+
+Use npm to install the project's dependencies inside the project.
+In your terminal, run the following command:
+
+[,bash]
+----
+npm ci
+----
+
+This command installs the dependencies listed in `package-lock.json` into the `node_modules/` directory inside the project.
+This directory should _not_ be committed to the source control repository.
+
+=== Use your local project
+
+If you want to use the project locally before it is published, you can specify the path to the extensions in the `local-antora-playbook.yml` file.
+
+[,yaml]
+----
+asciidoc:
+  attributes:
+  extensions:
+  - '<path-to-local-project>/antora-extensions-and-macros/extensions/<extension-name>'
+----

package/extensions/add-global-attributes.js

@@ -0,0 +1,77 @@
+/* Example use in the playbook
+* antora:
+    extensions:
+ *    - require: ./extensions/add-global-attributes.js
+       org: example
+       repo: test
+       branch: main
+*/
+
+const yaml = require('js-yaml');
+const { Octokit } = require("@octokit/rest");
+const { retry } = require("@octokit/plugin-retry");
+const OctokitWithRetries = Octokit.plugin(retry);
+const chalk = require('chalk')
+
+
+module.exports.register = function ({ config }) {
+  const logger = this.getLogger('global-attributes-extension')
+
+  this
+    .on('playbookBuilt', async ({ playbook }) => {
+
+      const globalAttributesUrl = `/repos/${config.org}/${config.repo}/contents/global-attributes?ref=${config.branch}`;
+
+      let githubOptions = {
+        userAgent: 'Redpanda Docs',
+        baseUrl: 'https://api.github.com',
+      };
+
+      if(process.env.GLOBAL_ATTRIBUTES_GITHUB_TOKEN){
+        githubOptions.auth = process.env.GLOBAL_ATTRIBUTES_GITHUB_TOKEN;
+      } else {
+        logger.warn('GLOBAL_ATTRIBUTES_GITHUB_TOKEN environment variable not set. Attempting unauthenticated request.')
+      }
+
+      const github = new OctokitWithRetries(githubOptions);
+
+      try {
+        // Request the contents of the directory from the GitHub API
+        const response = await github.request('GET ' + globalAttributesUrl);
+
+        if (response.status === 200) {
+          const directoryContents = response.data;
+
+          // Filter out only YAML files
+          const yamlFiles = directoryContents.filter(file => file.name.endsWith('.yaml') || file.name.endsWith('.yml'));
+
+          let globalAttributes = {};
+          for (let file of yamlFiles) {
+            const fileResponse = await github.request('GET ' + file.download_url);
+            const fileData = yaml.load(fileResponse.data);
+            globalAttributes = {...globalAttributes, ...fileData};
+          }
+
+          let mergedAttributes = {
+            ...globalAttributes,
+            ...playbook.asciidoc.attributes
+          };
+
+          playbook.asciidoc.attributes = mergedAttributes;
+
+          console.log(chalk.green('Merged global attributes into playbook'));
+
+        } else {
+          logger.warn(`Could not fetch global attributes: ${response.statusText}`);
+          return null
+        }
+      } catch(error) {
+        if (error.response.url) {
+          logger.warn(error.status + ' Could not get ' + error.response.url)
+        }
+        else {
+          logger.warn(error)
+        }
+      }
+    })
+}

package/extensions/algolia-indexer/generate-index.js

@@ -0,0 +1,209 @@
+'use strict'
+
+const { parse }   = require('node-html-parser')
+const { decode }  = require('html-entities')
+const path        = require('path')
+const URL         = require('url')
+const chalk       = require('chalk')
+
+/**
+ * Generates an Algolia index:
+ *
+ * Iterates over the specified pages and creates the indexes.
+ *
+ * @memberof algolia-indexer
+ *
+ * @param {Object} playbook - The configuration object for Antora.
+ * @param {Object} contentCatalog - The Antora content catalog, with pages and metadata.
+ * @param {Object} [config={}] - Configuration options
+ * @param {Boolean} config.indexLatestOnly - If true, only index the latest version of any given page.
+ * @param {Object} config.logger - Logger to use
+ * @typedef {Object} SearchIndexData
+ * @returns {SearchIndexData} A data object that contains the Algolia index
+ */
+function generateIndex (playbook, contentCatalog, { indexLatestOnly = false, excludes = [], logger } = {}) {
+  if (!process.env.ALGOLIA_ADMIN_API_KEY || !process.env.ALGOLIA_APP_ID || !process.env.ALGOLIA_INDEX_NAME) return
+  if (!logger) logger = process.env.NODE_ENV === 'test' ? { info: () => undefined } : console
+
+  const algolia = {}
+
+  console.log(chalk.cyan('Indexing...'))
+
+  // Select indexable pages
+  const pages = contentCatalog.getPages((page) => {
+    if (!page.out || page.asciidoc?.attributes?.noindex != null) return
+    return {}
+  })
+  if (!pages.length) return {}
+
+  // Handle the site URL
+  let siteUrl = playbook.site.url
+  if (!siteUrl) {
+    siteUrl = ''
+  }
+  if (siteUrl.charAt(siteUrl.length - 1) === '/') {
+    siteUrl = siteUrl.substr(0, siteUrl.length - 1)
+  }
+  const urlPath = extractUrlPath(siteUrl)
+
+  const documents = {}
+  var algoliaCount = 0
+
+  for (var i = 0; i < pages.length; i++) {
+    const page = pages[i]
+    const root = parse(
+      page.contents,
+      {
+        blockTextElements: {
+          code: true,
+        },
+      }
+    )
+
+    // skip pages marked as "noindex" for "robots"
+    const noindex = root.querySelector('meta[name=robots][content=noindex]')
+    if (noindex) {
+      continue
+    }
+
+    // Compute a flag identifying if the current page is in the
+    // "current" component version.
+    // When indexLatestOnly is set, we only index the current version.
+    const component = contentCatalog.getComponent(page.src.component)
+    const home = contentCatalog.getComponent('home')
+    const thisVersion = contentCatalog.getComponentVersion(component, page.src.version)
+    const latestVersion = component.latest
+    const isCurrent = thisVersion === latestVersion
+
+    if (indexLatestOnly && !isCurrent) continue
+
+    // capture the component name and version
+    const cname = component.name
+    const version = page.src.version
+
+    // handle the page keywords
+    const kw = root.querySelector('meta[name=keywords]')
+    var keywords = []
+    if (kw) {
+      keywords = kw.getAttribute('content')
+      keywords = keywords ? keywords.split(/,\s*/) : []
+    }
+
+    // gather page breadcrumbs
+    const breadcrumbs = []
+    root.querySelectorAll('nav.breadcrumbs > ul > li a')
+      .forEach((elem) => {
+        var url = path.resolve(
+          path.join('/', page.out.dirname),
+          elem.getAttribute('href')
+        )
+        breadcrumbs.push({
+          u: url,
+          t: elem.text
+        })
+      })
+
+    const images = {
+      'get started': 'get-started-icon.png',
+      'develop': 'develop-icon.png',
+      'deploy': 'deploy-icon.png',
+      'manage': 'manage-icon.png'
+    }
+
+    var image = {}
+
+    if (breadcrumbs.length > 1) {
+      const lowercaseBreadcrumb = breadcrumbs[1].t.toLowerCase()
+      for (let key in images) {
+        if (lowercaseBreadcrumb.includes(key)) {
+          image.src = `${home.url}_images/${images[key]}`
+          image.alt = key
+          break
+        }
+      }
+    }
+
+    // Start handling the article content
+    const article = root.querySelector('article.doc')
+    if (!article) {
+      logger.warn(`Page is not an article...skipping ${page.pub.url}`)
+      continue
+    }
+
+    // handle titles
+    const h1 = article.querySelector('h1')
+    if (!h1) {
+      logger.error(`No H1 in ${page.pub.url}`)
+      process.exit(1)
+    }
+    const documentTitle = h1.text
+    h1.remove()
+
+    const titles = []
+    article.querySelectorAll('h2,h3,h4,h5,h6').forEach((title) => {
+      var id = title.getAttribute('id')
+      if (id) {
+        titles.push({
+          t: title.text,
+          h: id,
+        })
+      }
+      title.remove()
+    })
+
+    // exclude elements within the article that should not be indexed
+    excludes.forEach((excl) => {
+      if (!excl) return
+      article.querySelectorAll(excl).map((e) =>  e.remove())
+    })
+
+    var intro = article.querySelector('p');
+    // decode any HTML entities
+    intro = decode(intro.rawText);
+
+    // establish structure in the Algolia index
+    if (!(cname in algolia)) algolia[cname] = {}
+    if (!(version in algolia[cname])) algolia[cname][version] = []
+
+    // Handle the article text
+    var text = decode(article.text)
+    text = text.replace(/\n/g, ' ')
+      .replace(/\r/g, ' ')
+      .replace(/\s+/g, ' ')
+      .trim()
+    if (text.length > 4000) text = text.substr(0, 4000)
+
+    var tag = `${component.title}-${version}`
+
+    const indexItem = {
+      title: documentTitle,
+      product: component.title,
+      version: version,
+      image: image? image: '',
+      text: text,
+      breadcrumbs: breadcrumbs,
+      intro: intro,
+      objectID: urlPath + page.pub.url,
+      titles: titles,
+      keywords: keywords,
+      _tags: [tag]
+    }
+
+    algolia[cname][version].push(indexItem)
+    algoliaCount++
+  }
+
+  return algolia
+}
+
+// Extract the path from a URL
+function extractUrlPath (url) {
+  if (url) {
+    if (url.charAt() === '/') return url
+    const urlPath = URL.parse(url).pathname
+    return urlPath === '/' ? '' : urlPath
+  }
+  return ''
+}
+
+module.exports = generateIndex

package/extensions/algolia-indexer/index.js

@@ -0,0 +1,83 @@
+'use strict'
+
+const generateIndex       = require('./generate-index')
+const chalk               = require('chalk')
+const algoliasearch       = require('algoliasearch');
+const fs                  = require('fs');
+const path                = require('path');
+
+/**
+ * Algolia indexing for an Antora documentation site.
+ *
+ * @module antora-algolia-indexer
+ */
+function register ({
+  config: {
+    indexLatestOnly,
+    excludes,
+    ...unknownOptions
+  }
+}) {
+  const logger = this.getLogger('algolia-indexer-extension')
+
+  if (!process.env.ALGOLIA_ADMIN_API_KEY || !process.env.ALGOLIA_APP_ID || !process.env.ALGOLIA_INDEX_NAME) return
+
+  var client;
+  var index;
+
+  // Connect and authenticate with Algolia
+  client = algoliasearch(process.env.ALGOLIA_APP_ID, process.env.ALGOLIA_ADMIN_API_KEY);
+
+  // Create a new index and add a record
+  index = client.initIndex(process.env.ALGOLIA_INDEX_NAME);
+
+  if (Object.keys(unknownOptions).length) {
+    const keys = Object.keys(unknownOptions)
+    throw new Error(`Unrecognized option${keys.length > 1 ? 's' : ''} specified: ${keys.join(', ')}`)
+  }
+
+  this.on('beforePublish', ({ playbook, siteCatalog, contentCatalog }) => {
+    const algolia = generateIndex(playbook, contentCatalog, { indexLatestOnly, excludes, logger })
+    // write the Algolia indexes
+    var algoliaCount = 0
+    Object.keys(algolia).forEach((c) => {
+      Object.keys(algolia[c]).forEach((v) => {
+        algoliaCount += algolia[c][v].length
+        // Save all records to the index
+        index.saveObjects(algolia[c][v]).wait();
+        siteCatalog.addFile({
+          mediaType: 'application/json',
+          contents: Buffer.from(
+            JSON.stringify(algolia[c][v])
+          ),
+          src: { stem: `algolia-${c}` },
+          out: { path: `algolia-${c}-${v}.json` },
+          pub: { url: `/algolia-${c}-${v}.json`, rootPath: '' },
+        })
+      })
+    })
+    index.setSettings({
+      attributesForFaceting: [
+        'version',
+        'product'
+      ]
+    })
+    console.log(`${chalk.bold(algoliaCount)} Algolia index entries created`)
+    // Get and print the count of all records in the index
+    let recordCount = 0;
+    index
+      .browseObjects({
+        query: '', // for all records
+        batch: batch => {
+          recordCount += batch.length;
+        }
+      })
+      .then(() => {
+        console.log('Total Records:', recordCount);
+      })
+      .catch(err => console.log(err));
+  })
+}
+
+module.exports = { generateIndex, register }
+

package/extensions/algolia-indexer/lazy-readable.js

@@ -0,0 +1,18 @@
+'use strict'
+
+const { PassThrough } = require('stream')
+
+// adapted from https://github.com/jpommerening/node-lazystream/blob/master/lib/lazystream.js | license: MIT
+class LazyReadable extends PassThrough {
+  constructor (fn, options) {
+    super(options)
+    this._read = function () {
+      delete this._read // restores original method
+      fn.call(this, options).on('error', this.emit.bind(this, 'error')).pipe(this)
+      return this._read.apply(this, arguments)
+    }
+    this.emit('readable')
+  }
+}
+
+module.exports = LazyReadable

package/extensions/algolia-indexer/multi-file-read-stream.js

@@ -0,0 +1,24 @@
+'use strict'
+
+const fs = require('fs')
+const { PassThrough } = require('stream')
+
+class MultiFileReadStream extends PassThrough {
+  constructor (paths) {
+    super()
+    ;(this.queue = this.createQueue(paths)).next()
+  }
+
+  * createQueue (paths) {
+    for (const path of paths) {
+      fs.createReadStream(path)
+        .once('error', (err) => this.destroy(err))
+        .once('end', () => this.queue.next())
+        .pipe(this, { end: false })
+      yield
+    }
+    this.push(null)
+  }
+}
+
+module.exports = MultiFileReadStream

package/extensions/algolia-indexer/template.js

@@ -0,0 +1,3 @@
+'use strict'
+
+module.exports = (string, vars) => string.replace(/\${([^}]+?)}/g, (_, name) => vars[name])

package/extensions/replace-attributes-in-attachments.js

@@ -0,0 +1,29 @@
+module.exports.register = function ({ config }) {
+  const { family = 'attachment' } = config;
+  const logger = this.getLogger('replace-attributes-in-attachments-extension');
+
+  const sanitizeAttributeValue = (value) => String(value).replace("@", "");
+
+  this.on('documentsConverted', ({playbook, contentCatalog}) => {
+    for (const { versions } of contentCatalog.getComponents()) {
+      for (const { name: component, version } of versions) {
+        if (component !== 'ROOT') continue;
+        const attachments = contentCatalog.findBy({ component, version, family });
+        for (const attachment of attachments) {
+          let contentString = String.fromCharCode(...attachment['_contents']);
+          const attributes = attachment.src.origin.descriptor.asciidoc.attributes;
+          const mergedAttributes = {
+            ...playbook.asciidoc.attributes,
+            ...attributes
+          };
+          for (const key in mergedAttributes) {
+            const placeholder = "{" + key + "}";
+            const sanitizedValue = sanitizeAttributeValue(mergedAttributes[key]);
+            contentString = contentString.replace(new RegExp(placeholder, 'g'), sanitizedValue);
+          }
+          attachment['_contents'] = Buffer.from(contentString, "utf-8");
+        }
+      }
+    }
+  });
+}

package/extensions/unlisted-pages.js

@@ -0,0 +1,38 @@
+module.exports.register = function ({ config }) {
+  const { addToNavigation, unlistedPagesHeading = 'Unlisted Pages' } = config
+  const logger = this.getLogger('unlisted-pages-extension')
+  this
+    .on('navigationBuilt', ({ contentCatalog }) => {
+      contentCatalog.getComponents().forEach(({ versions }) => {
+        versions.forEach(({ name: component, version, navigation: nav, url: defaultUrl }) => {
+          if (component === 'api') return;
+          const navEntriesByUrl = getNavEntriesByUrl(nav)
+          const unlistedPages = contentCatalog
+            .findBy({ component, version, family: 'page' })
+            .filter((page) => page.out)
+            .reduce((collector, page) => {
+              if ((page.pub.url in navEntriesByUrl) || page.pub.url === defaultUrl) return collector
+              logger.warn({ file: page.src, source: page.src.origin }, 'detected unlisted page')
+              return collector.concat(page)
+            }, [])
+          if (unlistedPages.length && addToNavigation) {
+            nav.push({
+              content: unlistedPagesHeading,
+              items: unlistedPages.map((page) => {
+                return { content: page.asciidoc.navtitle, url: page.pub.url, urlType: 'internal' }
+              }),
+              root: true,
+            })
+          }
+        })
+      })
+    })
+}
+
+function getNavEntriesByUrl (items = [], accum = {}) {
+  items.forEach((item) => {
+    if (item.urlType === 'internal') accum[item.url.split('#')[0]] = item
+    getNavEntriesByUrl(item.items, accum)
+  })
+  return accum
+}

package/extensions/version-fetcher/getLatestConsoleVersion.js

@@ -0,0 +1,33 @@
+// Fetch the latest release version from GitHub
+const { Octokit } = require("@octokit/rest");
+const { retry } = require("@octokit/plugin-retry");
+const OctokitWithRetries = Octokit.plugin(retry);
+const owner = 'redpanda-data';
+const repo = 'console';
+
+let githubOptions = {
+  userAgent: 'Redpanda Docs',
+  baseUrl: 'https://api.github.com',
+};
+
+if (process.env.REDPANDA_GITHUB_TOKEN) {
+  githubOptions.auth = process.env.REDPANDA_GITHUB_TOKEN;
+}
+
+const github = new OctokitWithRetries(githubOptions);
+
+var latestConsoleReleaseVersion;
+
+module.exports = async () => {
+  await github.rest.repos.getLatestRelease({
+    owner,
+    repo,
+  }).then((release => {
+    const tag = release.data.tag_name;
+    latestConsoleReleaseVersion = tag.replace('v','');
+  })).catch((error => {
+    console.error(error)
+    return null
+  }))
+  return latestConsoleReleaseVersion;
+};

package/extensions/version-fetcher/getLatestRedpandaVersion.js

@@ -0,0 +1,46 @@
+// Fetch the latest release version from GitHub
+const { Octokit } = require("@octokit/rest");
+const { retry } = require("@octokit/plugin-retry");
+const OctokitWithRetries = Octokit.plugin(retry);
+const owner = 'redpanda-data';
+const repo = 'redpanda';
+
+let githubOptions = {
+  userAgent: 'Redpanda Docs',
+  baseUrl: 'https://api.github.com',
+};
+
+if (process.env.REDPANDA_GITHUB_TOKEN) {
+  githubOptions.auth = process.env.REDPANDA_GITHUB_TOKEN;
+}
+
+const github = new OctokitWithRetries(githubOptions);
+
+var latestRedpandaReleaseVersion;
+var latestRedpandaReleaseCommitHash;
+
+module.exports = async () => {
+  await github.rest.repos.getLatestRelease({
+    owner,
+    repo,
+  }).then(async function(release) {
+    const tag = release.data.tag_name;
+    latestRedpandaReleaseVersion = tag.replace('v','');
+    await github.rest.git.getRef({
+      owner,
+      repo,
+      ref: `/tags/${tag}`,
+    }).then(async function(tagRef) {
+      const releaseSha = tagRef.data.object.sha;
+      await github.rest.git.getTag({
+        owner,
+        repo,
+        tag_sha: releaseSha,
+      }).then((tag => latestRedpandaReleaseCommitHash = tag.data.object.sha.substring(0, 7)))
+    })
+  }).catch((error => {
+    console.error(error)
+    return null
+  }))
+  return [latestRedpandaReleaseVersion, latestRedpandaReleaseCommitHash];
+};

package/extensions/version-fetcher/set-latest-version.js

@@ -0,0 +1,60 @@
+/* Example:
+antora:
+  extensions:
+  - require: ./extensions/setLatestVersion.js
+*/
+
+const GetLatestRedpandaVersion = require('./getLatestRedpandaVersion');
+const GetLatestConsoleVersion = require('./getLatestConsoleVersion');
+const chalk = require('chalk')
+
+
+module.exports.register = function ({ config }) {
+  const logger = this.getLogger('set-latest-version-extension')
+  if (!process.env.REDPANDA_GITHUB_TOKEN) {
+    logger.warn('REDPANDA_GITHUB_TOKEN environment variable not set. Attempting unauthenticated request.');
+  }
+  this
+    .on('playbookBuilt', async ({ playbook }) => {
+      try {
+        const LatestConsoleVersion = await GetLatestConsoleVersion();
+        if (!playbook.asciidoc) {
+          playbook.asciidoc = {};
+        }
+
+        if (!playbook.asciidoc.attributes) {
+          playbook.asciidoc.attributes = {};
+        }
+        playbook.asciidoc.attributes['latest-console-version'] = LatestConsoleVersion
+        console.log(`${chalk.green('Set Redpanda Console version to')} ${chalk.bold(LatestConsoleVersion)}`);
+      } catch(error) {
+        logger.warn(error)
+      }
+    })
+    .on('contentClassified', async ({ contentCatalog }) => {
+      try {
+        const LatestRedpandaVersion = await GetLatestRedpandaVersion();
+        const components = await contentCatalog.getComponents();
+        for (let i = 0; i < components.length; i++) {
+          let component = components[i];
+          if (LatestRedpandaVersion.length !== 2) logger.warn('Could not find the latest Redpanda versions - using defaults');
+          if (component.name === 'ROOT') {
+
+            if (!component.latest.asciidoc) {
+              component.latest.asciidoc = {};
+            }
+
+            if (!component.latest.asciidoc.attributes) {
+              component.latest.asciidoc.attributes = {};
+            }
+
+            component.latest.asciidoc.attributes['full-version'] = `${LatestRedpandaVersion[0]}`;
+            component.latest.asciidoc.attributes['latest-release-commit'] = `${LatestRedpandaVersion[1]}`;
+            console.log(`${chalk.green('Set Redpanda version to')} ${chalk.bold(LatestRedpandaVersion[0])} ${chalk.bold(LatestRedpandaVersion[1])}`)
+          }
+        }
+      } catch(error) {
+        logger.warn(error)
+      }
+    })
+}

package/macros/config-ref.js

@@ -0,0 +1,43 @@
+/* Example use in a page
+*
+* config_ref:myConfigValue,true,tunable-properties[]
+*
+* Example use in playbook
+*
+* asciidoc:
+    extensions:
+    - './macros/config-ref.js'
+*/
+
+const buildConfigReference = ({ configRef, isKubernetes, isLink, path }) => {
+  let ref = '';
+  if (isLink) {
+    if (isKubernetes) {
+      ref = `xref:reference:${path}.adoc#${configRef}[storage.tieredConfig.${configRef}]`;
+    } else {
+      ref = `xref:reference:${path}.adoc#${configRef}[${configRef}]`;
+    }
+  } else {
+    ref = isKubernetes ? `storage.tieredConfig.${configRef}` : `${configRef}`;
+  }
+  return ref;
+}
+
+function inlineConfigMacro(context) {
+  return function () {
+    this.process((parent, target, attrs) => {
+      const [configRef, isLink, path] = target.split(',');
+      const isKubernetes = parent.getDocument().getAttributes()['env-kubernetes'] !== undefined;
+      const content = buildConfigReference({ configRef, isKubernetes, isLink: isLink === 'true', path });
+      return this.createInline(parent, 'quoted', content);
+    });
+  }
+}
+
+function register (registry, context) {
+  registry.inlineMacro('config_ref', inlineConfigMacro(context));
+}
+
+module.exports.register = register;
+
+

package/macros/glossary.js

@@ -0,0 +1,180 @@
+'use strict'
+
+const $glossaryContexts = Symbol('$glossaryContexts')
+const { posix: path } = require('path')
+const chalk = require('chalk')
+
+module.exports.register = function (registry, config = {}) {
+
+  const vfs = adaptVfs()
+
+  function adaptVfs () {
+    function getKey (src) {
+      return `${src.version}@${src.component}`
+    }
+
+    const contentCatalog = config.contentCatalog
+    if (!contentCatalog[$glossaryContexts]) contentCatalog[$glossaryContexts] = {}
+    const glossaryContexts = contentCatalog[$glossaryContexts]
+    const key = getKey(config.file.src)
+    if (!glossaryContexts[key]) {
+      glossaryContexts[key] = {
+        gloss: [],
+        self: undefined,
+        dlist: undefined,
+      }
+    }
+    const context = glossaryContexts[key]
+    return {
+      getContext: () => context,
+    }
+  }
+
+  //characters to replace by '_' in generated idprefix
+  const IDRX = /[/ _.-]+/g
+
+  function termId (term) {
+    return '_glossterm_' + term.toLowerCase().replace(IDRX, '_')
+  }
+
+  function dlistItem (context, term, def) {
+    const id = termId(term)
+    term = `anchor:${id}[${term}]${term}`
+    const termItem = context.self.createListItem(context.dlist, term)
+    const defItem = context.self.createListItem(context.dlist, def)
+    return [[termItem], defItem]
+  }
+
+  function glossaryBlockMacro () {
+    return function () {
+      const self = this
+      self.named('glossary')
+      self.$option('format', 'short') //no target between glossary:: and [params]
+      // self.positionalAttributes(['name', 'parameters'])
+      self.process(function (parent, target, attributes) {
+        const context = vfs.getContext()
+        const dlist = self.createList(parent, 'dlist')
+        context.self = self
+        context.dlist = dlist
+        context.gloss
+          .forEach(({ term, def }) => {
+            dlist.blocks.push(dlistItem(context, term, def))
+          })
+        return dlist
+      })
+    }
+  }
+
+  const TRX = /(<[a-z]+)([^>]*>.*)/
+
+  function glossaryInlineMacro () {
+    return function () {
+      const self = this
+      self.named('glossterm')
+      //Specifying the regexp allows spaces in the term.
+      self.$option('regexp', /glossterm:([^[]+)\[(|.*?[^\\])\]/)
+      self.positionalAttributes(['definition'])
+      self.process(function (parent, target, attributes) {
+        const term = attributes.term || target
+        const document = parent.document
+        const context = vfs.getContext()
+        // See if a predefined list of terms is available
+        const globalTerms = document.getAttribute("terms") || [];
+        const localTerms = document.getAttribute("local-terms") || [];
+        let mergedTerms;
+        if(globalTerms.length > 0 && localTerms.length > 0){
+            // Convert globalTerms to a Map for fast look up
+            let globalTermsMap = new Map(globalTerms.map(i => [i.term, i]));
+
+            // Create a new array based on localTerms, but replace items if they exist in globalTerms
+            mergedTerms = localTerms.map(item => globalTermsMap.has(item.term) ? globalTermsMap.get(item.term) : item);
+
+            // Update localTerms with the mergedTerms
+            document.setAttribute("terms", mergedTerms);
+            console.log(chalk.green('Merged global terms with local terms'))
+        }
+        const termData = (mergedTerms || []).find((t) => t.term === term) || {};
+        const customLink = termData.link;
+        var tooltip = document.getAttribute('glossary-tooltip')
+        if (tooltip === 'true') tooltip = 'data-glossary-tooltip'
+        if (tooltip && tooltip !== 'title' && !tooltip.startsWith('data-')) {
+          console.log(`glossary-tooltip attribute '${tooltip}' must be 'true', 'title', or start with 'data-`)
+          tooltip = undefined
+        }
+        const logTerms = document.hasAttribute('glossary-log-terms')
+        var definition = termData.definition || attributes.definition
+        if (definition) {
+          logTerms && console.log(`${term}::  ${definition}`)
+          addItem(context, term, definition)
+        } else if (tooltip) {
+          const index = context.gloss.findIndex((candidate) => candidate.term === term)
+          definition = ~index ? context.gloss[index].def : `${term} not yet defined`
+        }
+        const links = document.getAttribute('glossary-links', 'true') === 'true'
+        var glossaryPage = document.getAttribute('glossary-page', '')
+        if (glossaryPage.endsWith('.adoc')) {
+          const page = config.contentCatalog.resolvePage(glossaryPage, config.file.src)
+          const relativizedPath = path.relative(path.dirname(config.file.pub.url), page.pub.url)
+          const prefix = attributes.prefix
+          glossaryPage = prefix ? [prefix, relativizedPath].join('/') : relativizedPath
+        }
+        const glossaryTermRole = document.getAttribute('glossary-term-role', 'glossary-term')
+        const attrs = glossaryTermRole ? { role: glossaryTermRole } : {}
+        const inline = links
+          ? customLink
+            ? self.createInline(parent, 'anchor', target, { type: 'link', target: customLink, attributes: attrs })
+            : self.createInline(parent, 'anchor', target, { type: 'xref', target: `${glossaryPage}#${termId(term)}`, reftext: target, attributes: attrs })
+          : self.createInline(parent, 'quoted', target, { attributes: attrs })
+        if (tooltip) {
+          const a = inline.convert()
+          const matches = a.match(TRX)
+          if (matches) {
+            return self.createInline(parent, 'quoted', `${matches[1]} ${tooltip}="${definition}"${matches[2]}`)
+          } else {
+            return self.createInline(parent, 'quoted', `<span ${tooltip}="${definition}">${a}</span>`)
+          }
+        }
+        return inline
+      })
+    }
+  }
+
+  function addItem (context, term, def) {
+    let i = 0
+    let comp = -1
+    for (; i < context.gloss.length && ((comp = term.localeCompare(context.gloss[i].term)) > 0); i++) {
+    }
+    if (comp < 0) {
+      context.gloss.splice(i, 0, { term, def })
+      if (context.self && context.dlist) {
+        context.dlist.blocks.splice(i, 0, dlistItem(context, term, def))
+      }
+    } else {
+      console.log(`duplicate glossary term ${term}`)
+    }
+  }
+
+  function doRegister (registry) {
+    if (typeof registry.blockMacro === 'function') {
+      registry.blockMacro(glossaryBlockMacro())
+    } else {
+      console.warn('no \'blockMacro\' method on alleged registry')
+    }
+    if (typeof registry.inlineMacro === 'function') {
+      registry.inlineMacro(glossaryInlineMacro())
+    } else {
+      console.warn('no \'inlineMacro\' method on alleged registry')
+    }
+  }
+
+  if (typeof registry.register === 'function') {
+    registry.register(function () {
+      //Capture the global registry so processors can register more extensions.
+      registry = this
+      doRegister(registry)
+    })
+  } else {
+    doRegister(registry)
+  }
+  return registry
+}

package/macros/helm-ref.js

@@ -0,0 +1,34 @@
+/* Example use in a page
+*
+* helm_ref:myConfigValue[]
+*
+* Example use in playbook
+*
+* asciidoc:
+    extensions:
+    - './macros/helm-ref.js'
+*/
+
+const buildConfigReference = ({ helmRef }) => {
+  let ref = '';
+  ref = helmRef ? `For default values and documentation for configuration options, see the https://artifacthub.io/packages/helm/redpanda-data/redpanda?modal=values&path=${helmRef}[values.yaml] file.` : `For default values and documentation for configuration options, see the https://artifacthub.io/packages/helm/redpanda-data/redpanda?modal=values[values.yaml] file.`;
+  return ref;
+}
+
+function inlineConfigMacro(context) {
+  return function () {
+    this.process((parent, target, attrs) => {
+      const [helmRef] = target.split(',');
+      const content = buildConfigReference({ helmRef });
+      return this.createInline(parent, 'quoted', content);
+    });
+  }
+}
+
+function register (registry, context) {
+  registry.inlineMacro('helm_ref', inlineConfigMacro(context));
+}
+
+module.exports.register = register;
+
+

package/package.json

@@ -0,0 +1,32 @@
+{
+  "name": "@redpanda-data/docs-extensions-and-macros",
+  "version": "2.5.0",
+  "description": "Antora extensions and macros developed for Redpanda documentation.",
+  "keywords": ["antora", "extension", "macro", "documentation", "redpanda"],
+  "author": {
+    "name": "Redpanda Docs Team"
+  },
+  "contributors": [
+    {
+      "name": "JakeSCahill",
+      "email": "jake@redpanda.com"
+    }
+  ],
+  "license": "ISC",
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/redpanda-data/docs-extensions-and-macros"
+  },
+  "dependencies": {
+    "@antora/site-generator": "3.1.2",
+    "@octokit/plugin-retry": "^4.1.3",
+    "@octokit/rest": "^19.0.7",
+    "algoliasearch": "^4.17.0",
+    "chalk": "4.1.2",
+    "js-yaml": "^4.1.0",
+    "gulp": "^4.0.2",
+    "gulp-connect": "^5.7.0",
+    "html-entities": "2.3",
+    "node-html-parser": "5.4.2-0"
+  }
+}