metanorma-release 0.2.1

data/README.adoc

@@ -0,0 +1,430 @@
+= metanorma-release
+:toc: macro
+:toclevels: 3
+
+Release lifecycle management for Metanorma documents.
+
+toc::[]
+
+== Overview
+
+`metanorma-release` manages the full release lifecycle of Metanorma documents:
+
+**Release** (producer side)::
+Discover compiled documents -> extract metadata from RXL -> detect changes -> package as zip -> publish to a platform (GitHub Releases, local filesystem).
+
+**Aggregate** (consumer side)::
+Discover repositories -> fetch published releases -> filter by channel and stage -> extract zip assets -> generate `index.json` with a file tree for any site generator.
+
+The output is platform-agnostic: a directory containing `index.json` and a tree of document files. Any site generator (Jekyll, Hugo, Vite) consumes that output independently.
+
+== Installation
+
+Add to your Gemfile:
+
+[source,ruby]
+----
+gem "metanorma-release"
+----
+
+Or install directly:
+
+[source,sh]
+----
+gem install metanorma-release
+----
+
+Requires Ruby >= 3.2.
+
+== Quick start
+
+=== CLI
+
+The gem ships three commands:
+
+[source,sh]
+----
+# Package compiled documents as zip archives
+mn-release package --output-dir _site --manifest metanorma.release.yml
+
+# Package and publish to a platform
+mn-release publish --platform github --output-dir _site --token $GITHUB_TOKEN
+
+# Aggregate published releases into a file tree + index.json
+mn-release aggregate --source github --organizations my-org --output-dir _site/cc
+----
+
+=== Rake tasks
+
+Register tasks in your `Rakefile`:
+
+[source,ruby]
+----
+require "metanorma/release/rake_tasks"
+
+Metanorma::Release::RakeTasks.install do |config|
+  config.output_dir = "_site"
+  config.manifest = "metanorma.release.yml"
+  config.platform = "github"
+end
+----
+
+This provides:
+
+* `rake mn:package` -- package compiled documents
+* `rake mn:publish` -- package and publish documents
+* `rake mn:aggregate` -- aggregate published releases
+
+=== Ruby API
+
+Use the pipelines directly for fine-grained control:
+
+[source,ruby]
+----
+deps = Metanorma::Release::ReleasePipeline::Dependencies.new(
+  extractor: Metanorma::Release::RxlExtractor.new,
+  filters: [],
+  change_detector: Metanorma::Release::ContentHashChangeDetector.new(previous_releases: {}),
+  packager: Metanorma::Release::ZipPackager.new,
+  publisher: Metanorma::Release::PlatformFactory.build_publisher("null", {}),
+  naming_registry: Metanorma::Release::NamingRegistry.default_registry,
+  manifest: nil,
+  channel_override: nil,
+  channel_config: nil
+)
+
+config = Metanorma::Release::ReleasePipeline::Config.new(
+  output_dir: "_site",
+  manifest_path: nil,
+  force: false,
+  force_replace_patterns: nil,
+  concurrency: 4,
+  default_visibility: "public"
+)
+
+result = Metanorma::Release::ReleasePipeline.new(deps).run(config)
+result.released  # => [#<DocumentMetadata ...>]
+result.skipped   # => [#<DocumentMetadata ...>]
+result.failed    # => [{ document: ..., error: "..." }]
+----
+
+== CLI reference
+
+=== `mn-release package`
+
+Package compiled documents into zip archives without publishing.
+
+[source,sh]
+----
+mn-release package [options]
+----
+
+[cols="1m,3",options="header"]
+|===
+|Option |Description
+|`--output-dir DIR` |Directory containing compiled documents (default: `_site`)
+|`--dest DIR` |Destination for zip packages (default: `dist`)
+|`--manifest FILE` |Release manifest file (default: `metanorma.release.yml`)
+|`--config SOURCE` |Channel config file or platform ref
+|===
+
+=== `mn-release publish`
+
+Package and publish documents to a platform.
+
+[source,sh]
+----
+mn-release publish [options]
+----
+
+[cols="1m,3",options="header"]
+|===
+|Option |Description
+|`--platform NAME` |Target platform: `github`, `local` (default: `github`)
+|`--output-dir DIR` |Compiled docs directory (default: `_site`)
+|`--manifest FILE` |Release manifest file (default: `metanorma.release.yml`)
+|`--force` |Force release even if unchanged
+|`--force-replace PAT` |Glob pattern for forced replacement (repeatable)
+|`--channels CHANS` |Override channels (comma-separated)
+|`--concurrency N` |Parallel workers (default: 4)
+|`--token TOKEN` |Platform auth token
+|`--config SOURCE` |Channel config file or platform ref
+|===
+
+=== `mn-release aggregate`
+
+Aggregate published releases from multiple repositories into a unified file tree.
+
+[source,sh]
+----
+mn-release aggregate [options]
+----
+
+[cols="1m,3",options="header"]
+|===
+|Option |Description
+|`--source SOURCE` |Discovery source: `github`, `local:PATH` (default: `github`)
+|`--organizations ORGS` |Comma-separated organization list
+|`--topic TOPIC` |Repository topic filter (default: `metanorma-release`)
+|`--repos REPOS` |Explicit repo list (comma-separated)
+|`--channels CHANS` |Filter channels (comma-separated)
+|`--stages STAGES` |Filter stages (comma-separated)
+|`--output-dir DIR` |Output directory (default: `_site/cc`)
+|`--file-routing MODE` |File layout: `by-document`, `flat`, `by-format` (default: `by-document`)
+|`--cache-dir DIR` |Cache directory for delta state
+|`--[no-]include-drafts` |Include draft releases
+|`--concurrency N` |Parallel repos (default: 4)
+|`--min-documents N` |Fail if fewer documents found (default: 0)
+|`--token TOKEN` |Platform auth token
+|===
+
+== Concepts
+
+=== Channels
+
+A channel is an `audience/category` pair that controls who can access a document:
+
+[source,ruby]
+----
+channel = Metanorma::Release::Channel.parse("public/standards")
+channel.public?   # => true
+channel.audience  # => "public"
+channel.category  # => "standards"
+----
+
+Audiences: `public`, `members`, `internal`. When omitted, audience defaults to `public`.
+
+=== Channel configuration
+
+A channel config defines the set of allowed channels for a project or organization, along with default visibility. This lets you enforce a channel taxonomy across all documents.
+
+.Config resolution order
+[arabic]
+. `--config` CLI flag (highest priority)
+. `config:` key in the release manifest
+. Directory walk: `.metanorma.yml`, `.metanorma.yaml`, or `.metanorma/channels.yml`
+. No config -- all channels allowed
+
+==== Config file format
+
+[source,yaml]
+----
+# .metanorma.yml
+channels:
+  - public/standards
+  - public/reports
+  - members/early-access
+  - internal/working-drafts
+defaults:
+  visibility: public
+  channels:
+    - public/standards
+----
+
+The `channels` list defines the taxonomy -- only these channels are valid. The `defaults` section sets fallback visibility and channels when a document doesn't match any manifest entry.
+
+==== Specifying config in the manifest
+
+Add a `config` key to `metanorma.release.yml`:
+
+[source,yaml]
+----
+config: local:/path/to/config.yml
+defaults:
+  visibility: public
+documents:
+  - source: sources/cc-18011.adoc
+    channels:
+      - public/standards
+----
+
+The config source can be:
+
+* `local:/path/to/config.yml` -- local file path
+* `myorg/myrepo` -- GitHub repo (reads `channels.yml` from root)
+* `myorg/myrepo#path/to/config.yml` -- GitHub repo with explicit path
+
+==== Ruby API
+
+[source,ruby]
+----
+# Parse from YAML
+config = Metanorma::Release::ChannelConfig.from_yaml(File.read(".metanorma.yml"))
+
+# Permissive config (all channels allowed)
+config = Metanorma::Release::ChannelConfig.empty
+
+# Validate a channel
+config.registry.valid?(Channel.parse("public/standards"))  # => true
+config.registry.valid?(Channel.parse("public/secret"))     # => false
+
+# Locate config by walking up from a directory
+config = Metanorma::Release::ConfigLocator.find("/path/to/project")
+----
+
+=== Naming strategies
+
+Tag and file naming varies by document type. Strategies are resolved via a registry:
+
+[cols="1m,1,2",options="header"]
+|===
+|Document type |Strategy |Tag format
+|standard (default) |`EditionNaming` |`cc-18011/ed1`
+|IETF draft |`InternetDraftNaming` |`id-ietf-foo/1`
+|IETF RFC |`RfcNaming` |`rfc-1234/ed1`
+|IEEE |`DraftSuffixNaming` |`ieee-8021/d1`
+|IHO, OGC |`VersionNaming` |`iho-s44/v1`
+|===
+
+Register custom strategies:
+
+[source,ruby]
+----
+registry = Metanorma::Release::NamingRegistry.default_registry
+registry.register("my-type", MyCustomNaming.new)
+----
+
+=== File routing
+
+The aggregation pipeline supports three file layout modes:
+
+[cols="1m,3",options="header"]
+|===
+|Mode |Example path
+|`by-document` (default) |`cc-18011/cc-18011.html`
+|`flat` |`cc-18011.html`
+|`by-format` |`html/cc-18011.html`
+|===
+
+=== Release manifest
+
+A `metanorma.release.yml` file controls which documents are published and to which channels:
+
+[source,yaml]
+----
+config: myorg/.metanorma
+defaults:
+  visibility: public
+  channels:
+    - public/standards
+documents:
+  - source: sources/cc-18011.adoc
+    channels:
+      - public/standards
+  - source: sources/cc-19060.adoc
+    visibility: members
+    channels:
+      - members/early-access
+  - pattern: "sources/draft-*.adoc"
+    channels:
+      - internal/working-drafts
+    stages:
+      - working-draft
+      - committee-draft
+----
+
+Documents not listed in the manifest use the `defaults` section. If no manifest exists, all documents are released as `public/standards`.
+
+Key fields:
+
+[cols="1m,3",options="header"]
+|===
+|Field |Description
+|`source` |Exact path match (highest priority)
+|`pattern` |Glob pattern match
+|`visibility` |`public`, `members`, or `private`
+|`channels` |List of target channels
+|`stages` |Allow-list of document stages
+|`config` |Channel config source (see <<channel-configuration>>)
+|===
+
+=== Value objects
+
+All domain types are immutable, frozen, and use value-based equality:
+
+* `DocumentId` -- normalized document identifier (`CC 18011` -> `cc-18011`)
+* `DocumentVersion` -- edition + stage + pre-release flag
+* `DocumentStage` -- published, draft, working-draft, committee-draft, etc.
+* `Channel` -- audience/category pair
+* `ReleaseTag` -- tag string with pre-release flag
+* `ContentHash` -- SHA-256 content fingerprint
+* `RepoRef` -- owner/repo reference
+
+=== Bibliography enrichment
+
+`RelatonEnricher` generates `index.json` and `index.yaml` from RXL (Relaton XML) files found in aggregated documents. It auto-detects the Relaton flavor from document metadata:
+
+[source,ruby]
+----
+enricher = Metanorma::Release::RelatonEnricher.new(flavor: "calconnect")
+result = enricher.enrich(document_index, output_dir)
+# writes: output_dir/relaton/index.json
+#         output_dir/relaton/index.yaml
+----
+
+Flavor detection tries these gems in order: `relaton-calconnect`, `relaton-iso`, `relaton-iec`, `relaton-ogc`, `relaton-ietf`, and others. If a flavor gem is not installed, it falls back to `Relaton::Bib::Item` from the `relaton-bib` runtime dependency.
+
+== Architecture
+
+=== Dependency flow
+
+Unidirectional, no cycles:
+
+----
+domain/  ->  release/  ->  platform/
+         ->  aggregation/ ->  platform/
+                          ->  cli/
+----
+
+* `domain/` has zero knowledge of pipelines, platforms, or CLI
+* Pipelines depend on domain + interfaces, not platform implementations
+* Platform adapters depend on interfaces + domain, not pipelines
+* CLI delegates to command classes; commands depend on pipelines + platform factory
+* Commands use `ConfigResolver` mixin for channel config resolution
+
+=== Patterns
+
+Value Objects:: Immutable, frozen, value-based equality via `eql?`/`hash`. All fields included in equality comparison.
+
+Strategy Pattern:: Pluggable algorithms resolved via registry. Adding a new document type or platform requires zero changes to existing code.
+
+Pipeline with DI:: Pipelines receive all dependencies through constructors. No global state, no service locators.
+
+Null Object:: Disabled features inject null implementations (`NullDeltaState`, `NullPublisher`, `NullCacheStore`) instead of adding conditional checks.
+
+Result Types:: Pipelines return frozen Structs. Errors are collected, not raised. The caller decides whether to abort.
+
+Command Pattern:: CLI delegates to `PackageCommand`, `PublishCommand`, and `AggregateCommand` classes. Each command encapsulates pipeline construction and configuration resolution via the `ConfigResolver` mixin.
+
+=== Extending
+
+|===
+|To add... |Do this
+
+|A new platform
+|Create a directory under `platform/` with `Publisher`, `Discoverer`, `Fetcher`, `ManifestReader` classes; register in `PlatformFactory`
+
+|A new naming strategy
+|Create a class that includes `NamingStrategy`; register via `NamingRegistry#register`
+
+|A new file routing mode
+|Create a class with `#compute_path(file_name, metadata)`; register in `FileRoutingFactory`
+
+|A new filter
+|Create a class that includes `Filter`; pass to the pipeline's `filters` array
+
+|A new channel config source
+|Create a class that includes `ConfigFetcher` with a `#fetch(source)` method
+|===
+
+== Development
+
+[source,sh]
+----
+bundle install
+bundle exec rspec
+----
+
+== License
+
+BSD-2-Clause. See `LICENSE` for details.

data/Rakefile

@@ -0,0 +1,8 @@
+# frozen_string_literal: true
+
+require 'bundler/gem_tasks'
+require 'rspec/core/rake_task'
+
+RSpec::Core::RakeTask.new(:spec)
+
+task default: :spec

data/exe/mn-release

@@ -0,0 +1,6 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require 'metanorma/release/cli'
+
+Metanorma::Release::CLI.run(ARGV)

data/lib/metanorma/release/aggregation_interfaces.rb

@@ -0,0 +1,33 @@
+# frozen_string_literal: true
+
+module Metanorma
+  module Release
+    module RepoDiscoverer
+      def discover
+        raise NotImplementedError, "#{self.class} must implement #discover"
+      end
+    end
+
+    module ReleaseFetcher
+      def fetch(repo, etag: nil)
+        raise NotImplementedError, "#{self.class} must implement #fetch"
+      end
+    end
+
+    module ManifestReader
+      def read(repo)
+        raise NotImplementedError, "#{self.class} must implement #read"
+      end
+    end
+
+    module IndexGenerator
+      def generate(documents, output_dir, format:, parameters:)
+        raise NotImplementedError, "#{self.class} must implement #generate"
+      end
+    end
+
+    FetchResult = Struct.new(:releases, :etag, :unchanged?, keyword_init: true)
+    RepoReport  = Struct.new(:releases, :included, :skipped, :reason, :errors, keyword_init: true)
+    RepoError   = Struct.new(:tag, :message, keyword_init: true)
+  end
+end

data/lib/metanorma/release/aggregation_pipeline.rb

@@ -0,0 +1,155 @@
+# frozen_string_literal: true
+
+module Metanorma
+  module Release
+    class AggregationPipeline
+      Dependencies = Struct.new(
+        :discoverer, :fetcher, :manifest_reader,
+        :channel_filter, :stage_filter,
+        :asset_processor, :delta_state,
+        keyword_init: true
+      )
+
+      Config = Struct.new(
+        :organizations, :channels, :topic,
+        :concurrency, :include_drafts, :fail_on_error,
+        keyword_init: true
+      )
+
+      Result = Struct.new(
+        :documents, :repo_count, :channels_found,
+        :report, :failed_repos,
+        keyword_init: true
+      )
+
+      def initialize(deps)
+        @deps = deps
+      end
+
+      def run(config, output_dir)
+        @deps.delta_state.load
+        repos = @deps.discoverer.discover
+        documents = []
+        reports = []
+        failed_repos = []
+
+        repos.each do |repo|
+          repo_docs, report = process_repo(repo, output_dir, config)
+          documents.concat(repo_docs)
+          reports << report
+        rescue StandardError => e
+          failed_repos << RepoError.new(tag: repo.to_s, message: e.message)
+          raise if config.fail_on_error
+        end
+
+        @deps.delta_state.save
+
+        Result.new(
+          documents: documents,
+          repo_count: repos.length,
+          channels_found: documents.flat_map { |d| d.channels || [] }.uniq.sort,
+          report: reports,
+          failed_repos: failed_repos
+        )
+      end
+
+      private
+
+      def process_repo(repo, _output_dir, config)
+        repo_key = repo.to_s
+
+        manifest_channels = @deps.manifest_reader.read(repo)
+        if manifest_channels && !@deps.channel_filter.overlaps?(manifest_channels)
+          return [], RepoReport.new(releases: 0, included: 0, skipped: 0,
+                                    reason: 'channel manifest', errors: [])
+        end
+
+        etag = @deps.delta_state.etag(repo_key)
+        fetch_result = @deps.fetcher.fetch(repo, etag: etag)
+
+        if fetch_result.unchanged?
+          return [], RepoReport.new(releases: 0, included: 0, skipped: 0,
+                                    reason: 'etag unchanged', errors: [])
+        end
+
+        current_tags = []
+        documents = []
+        errors = []
+
+        fetch_result.releases.each do |release|
+          metadata = ReleaseMetadata.from_release_body(release.body)
+          next if metadata.nil?
+
+          next unless @deps.channel_filter.matches?(metadata.to_h)
+          next unless @deps.stage_filter.matches?(metadata.to_h)
+          next if release.prerelease && !config.include_drafts
+
+          tag = release.tag_name
+          current_tags << tag
+
+          content_hash = extract_content_hash(release.body)
+
+          if @deps.delta_state.processed?(repo_key, tag, content_hash)
+            files = @deps.delta_state.release_files(repo_key, tag)
+            documents << build_document(metadata, files, content_hash, release, repo)
+            next
+          end
+
+          zip_asset = find_zip_asset(release)
+          next unless zip_asset
+
+          result = @deps.asset_processor.process(zip_asset.data, metadata.to_h)
+          @deps.delta_state.mark_processed(repo_key, tag, content_hash, result.files.map(&:path))
+          documents << build_document(metadata, result.files.map(&:path), content_hash, release, repo)
+        rescue StandardError => e
+          errors << RepoError.new(tag: release.tag_name, message: e.message)
+        end
+
+        @deps.delta_state.cleanup_stale(repo_key, current_tags)
+        @deps.delta_state.set_etag(repo_key, fetch_result.etag)
+
+        [documents, RepoReport.new(
+          releases: fetch_result.releases.length,
+          included: documents.length,
+          skipped: fetch_result.releases.length - documents.length,
+          reason: nil, errors: errors
+        )]
+      end
+
+      def build_document(metadata, files, content_hash, release, repo)
+        source = DocumentSource.new(
+          owner: repo.owner, repo: repo.repo,
+          tag: release.tag_name,
+          release_url: release.html_url,
+          release_date: release.published_at
+        )
+
+        file_structs = files.map { |f| DocumentFile.new(name: File.basename(f), path: f) }
+
+        AggregatedDocument.new(
+          id: metadata.id, title: metadata.title,
+          edition: metadata.edition, stage: metadata.stage,
+          doctype: metadata.doctype,
+          channels: metadata.channels,
+          formats: metadata.formats,
+          flavor: metadata.flavor,
+          content_hash: content_hash.to_s,
+          source: source, files: file_structs
+        )
+      end
+
+      def extract_content_hash(body)
+        return nil if body.nil?
+
+        match = body.match(/^content-hash:([a-f0-9]+)/)
+        match ? match[1] : nil
+      end
+
+      def find_zip_asset(release)
+        return nil unless release.assets
+
+        release.assets.find { |a| a.name.end_with?('.zip') }
+      end
+    end
+  end
+end

data/lib/metanorma/release/asset_processor.rb

@@ -0,0 +1,58 @@
+# frozen_string_literal: true
+
+begin
+  require 'zip'
+rescue LoadError
+  raise LoadError, "The rubyzip gem is required for AssetProcessor. Add `gem 'rubyzip'` to your Gemfile."
+end
+
+module Metanorma
+  module Release
+    class AssetProcessor
+      ProcessResult = Struct.new(:files, :channels, keyword_init: true)
+
+      CANONICALIZE_PATTERN = /-ed\d+(\.\d+)?-/
+
+      def initialize(output_dir:, routing:, canonicalize: true)
+        @output_dir = output_dir
+        @routing = routing
+        @canonicalize = canonicalize
+      end
+
+      def process(zip_data, metadata)
+        files = []
+
+        Dir.mktmpdir do |tmp_dir|
+          zip_path = File.join(tmp_dir, 'archive.zip')
+          File.binwrite(zip_path, zip_data)
+
+          Zip::File.open(zip_path) do |zip_file|
+            zip_file.each do |entry|
+              next if entry.directory?
+
+              raw_name = File.basename(entry.name)
+              file_name = @canonicalize ? canonicalize_name(raw_name) : raw_name
+              relative_path = @routing.compute_path(file_name, metadata)
+              dest_path = File.join(@output_dir, relative_path)
+
+              FileUtils.mkdir_p(File.dirname(dest_path))
+              entry.extract(dest_path) { true }
+
+              files << DocumentFile.new(name: file_name, path: relative_path)
+            end
+          end
+        end
+
+        ProcessResult.new(files: files, channels: metadata['channels'])
+      end
+
+      private
+
+      def canonicalize_name(name)
+        # Strip edition suffix: -ed1. → ., -ed1-wd. → -wd.
+        name.sub(/-ed\d+(\.\d+)?-(?=[a-z0-9])/, '-')
+            .sub(/-ed\d+(\.\d+)?\./, '.')
+      end
+    end
+  end
+end