metanorma-cli 1.16.4 → 1.16.5

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: dbf7c76fc0597e5829f5abd6f076334f5cd750fbdecb6b17e98a29ff038e0d6e
-  data.tar.gz: 40c5b5f7b6bcc3eab6fa010e937cb711e1c92eeee66954497bb7b2aa3ce4b2e8
+  metadata.gz: c0a53a754776e569091e1b9311196c7981651098eb6a7c0add1a44d11e93b127
+  data.tar.gz: 91c87751dc666c8e691ff14bc9a5251894e5690927dfcddcf102cc2dbaa6aecc
 SHA512:
-  metadata.gz: 189f44e361d625494d4169346d3426b22ea95f7b5c2754a2ed746a748fea440c04d562a1d17ca34ead364496d9fcfd5cb3d7edbfe0384db8dbcfe5fb6543c23d
-  data.tar.gz: e115f84be420250c35811a6a815b22fd2120c5bbe33fb1fee127f252c504d8e1e4ada9d6078e9e37baa304ff921cd6f64d62b07465d567151bc3f6eda599d426
+  metadata.gz: c9dcc6876d000cdd70debfe29d24a99d1b4af74e680bd90f372bef6c62ba88fe6863a366ad190cca1a28a8facd94bc25bbf2b4366c8f8099bf9fab744a5810c8
+  data.tar.gz: 275d15fe27f5e4b6663dc3aead1b7a767e162430a55b2e1a8692ad7fb1839db7abe60c3fb7fcef5ae1cabbcda700c0d4924e28ee82124c73f6c856020af1de69

data/docs/testing.adoc

@@ -0,0 +1,379 @@
+= Testing PRs with Gemfile.devel
+:toc:
+:toclevels: 3
+
+== Overview
+
+The `Gemfile.devel` mechanism allows you to test Pull Requests (PRs) of dependent gems against the full set of document samples and templates in metanorma-cli's CI workflow. This is critical for verifying that infrastructure changes to base gems don't break downstream flavors before releasing them.
+
+== The Problem
+
+Metanorma's architecture consists of a layered dependency tree of gems:
+
+* **Base layer**: `isodoc` provides core document processing
+* **Middle layer**: `metanorma-standoc` builds on isodoc, and flavor gems build on standoc
+* **Top layer**: Specialized flavors that depend on middle-layer flavors
+
+When you make a change to a base gem like `isodoc` or `metanorma-standoc`, the gem's own test suite may pass, but the change could break any of the 20+ downstream flavor gems. Without integration testing, these breaks only get discovered after release when users report crashes.
+
+== The Solution
+
+The CI workflow for metanorma-cli tests against sample documents from all metanorma flavor repositories. By creating a `Gemfile.devel` file in your metanorma-cli PR, you can specify which gem PRs/branches to use during these tests, enabling full integration testing before any releases.
+
+== How It Works
+
+=== Workflow Integration
+
+The reusable workflow at `metanorma/ci/.github/workflows/mn-processor-rake.yml` includes special handling for `Gemfile.devel`:
+
+1. When testing samples/templates, the workflow checks out the metanorma-cli repository (with your PR)
+2. It checks out each sample repository (e.g., `mn-samples-iso`, `mn-samples-ogc`)
+3. In the sample's directory, it modifies the Gemfile to:
+   - Add metanorma-cli from the local path (`../`)
+   - Append `eval_gemfile("../Gemfile.devel") rescue nil`
+4. When `bundle install` runs, it loads your gem overrides from Gemfile.devel
+5. The tests run using your specified gem versions
+
+=== Local Gemfile Support
+
+The main `Gemfile` in metanorma-cli also loads `Gemfile.devel` if it exists:
+
+[source,ruby]
+----
+begin
+  eval_gemfile("Gemfile.devel")
+rescue StandardError
+  nil
+end
+----
+
+This means `Gemfile.devel` works both in CI and in local development.
+
+== Usage Guide
+
+=== Basic Workflow
+
+1. **Create your Gemfile.devel**:
++
+[source,bash]
+----
+cp Gemfile.devel.example Gemfile.devel
+----
+
+2. **Edit the file to specify gem overrides**:
++
+[source,ruby]
+----
+# Test a PR branch of isodoc
+gem "isodoc", github: "metanorma/isodoc", branch: "feature/svg-conform"
+
+# Test main branch of metanorma-standoc
+gem "metanorma-standoc", github: "metanorma/metanorma-standoc", branch: "main"
+----
+
+3. **Commit and push your PR**:
++
+[source,bash]
+----
+git add Gemfile.devel
+git commit -m "Test isodoc PR #123 against samples"
+git push origin your-branch
+----
+
+4. **Monitor the CI workflow**:
+   - The `test-samples` and `test-templates` jobs will use your gem specifications
+   - Check the logs to verify the correct gems are being loaded
+   - Review any test failures to ensure compatibility
+
+=== Verifying Gem Loading
+
+To confirm that your gem override is being used, you can add diagnostic output to your gem PR. For example, in an isodoc PR:
+
+[source,ruby]
+----
+# In lib/isodoc.rb or similar initialization file
+warn "DEBUG: Loading isodoc from CUSTOM BRANCH for testing"
+----
+
+When the CI runs, this message will appear in the GitHub Actions logs, confirming the override worked.
+
+== Dependency Tree Reference
+
+Understanding the dependency tree helps you determine which gems need to be overridden:
+
+----
+isodoc (base layer)
+  └─ metanorma-standoc
+      ├─ metanorma-generic
+      │   ├─ metanorma-csa
+      │   ├─ metanorma-cc
+      │   ├─ metanorma-iho
+      │   └─ metanorma-ribose
+      ├─ metanorma-iso
+      │   ├─ metanorma-jis
+      │   ├─ metanorma-iec
+      │   └─ metanorma-bsi
+      ├─ metanorma-ieee
+      ├─ metanorma-itu
+      ├─ metanorma-nist
+      └─ metanorma-ogc
+
+Special cases:
+  - metanorma-bipm depends on: isodoc, standoc, generic, AND iso
+  - metanorma-plateau depends on: isodoc, standoc, iso, AND jis
+----
+
+== Common Testing Scenarios
+
+=== Scenario 1: Testing a Base Infrastructure Change
+
+You've made a change to `isodoc` that affects all downstream gems:
+
+[source,ruby]
+----
+# Gemfile.devel
+gem "isodoc", github: "metanorma/isodoc", branch: "feature/new-rendering"
+----
+
+This tests your isodoc PR against ALL flavor samples (ISO, ITU, OGC, BSI, etc.).
+
+=== Scenario 2: Testing Related PRs Together
+
+You have PRs in both `isodoc` and `metanorma-standoc` that work together:
+
+[source,ruby]
+----
+# Gemfile.devel
+gem "isodoc", github: "metanorma/isodoc", branch: "feature/base-api"
+gem "metanorma-standoc", github: "metanorma/metanorma-standoc", branch: "feature/use-base-api"
+----
+
+This ensures the two PRs are compatible before merging either.
+
+=== Scenario 3: Testing a Flavor-Specific Change
+
+You're fixing a bug in `metanorma-bsi` that also requires changes to `metanorma-iso`:
+
+[source,ruby]
+----
+# Gemfile.devel
+gem "metanorma-iso", github: "metanorma/metanorma-iso", branch: "fix/bsi-compatibility"
+gem "metanorma-bsi", github: "metanorma/metanorma-bsi", branch: "fix/rendering-issue"
+----
+
+=== Scenario 4: Testing Against main Branches (Pre-release Testing)
+
+Before releasing `isodoc`, verify that the main branch works with all samples:
+
+[source,ruby]
+----
+# Gemfile.devel
+gem "isodoc", github: "metanorma/isodoc", branch: "main"
+gem "metanorma-standoc", github: "metanorma/metanorma-standoc", branch: "main"
+----
+
+This is the "hotfix test" scenario from issue #370: test unreleased changes across the entire stack.
+
+=== Scenario 5: Local Development Testing
+
+When working across multiple repositories locally:
+
+[source,ruby]
+----
+# Gemfile.devel
+gem "isodoc", path: "../isodoc"
+gem "metanorma-standoc", path: "../metanorma-standoc"
+gem "metanorma-iso", path: "../metanorma-iso"
+----
+
+Run `bundle exec metanorma` locally to test your changes.
+
+== Advanced Usage
+
+=== Testing Specific PR Numbers
+
+You can reference pull requests directly using GitHub refs:
+
+[source,ruby]
+----
+# Test PR #123 from metanorma/isodoc
+gem "isodoc", github: "metanorma/isodoc", ref: "refs/pull/123/head"
+----
+
+=== Testing Forks
+
+If the PR is from a fork:
+
+[source,ruby]
+----
+gem "isodoc", github: "username/isodoc", branch: "feature-name"
+----
+
+=== Overriding Private Gems
+
+For gems hosted in private repositories (assuming you have access):
+
+[source,ruby]
+----
+gem "metanorma-bsi", github: "metanorma/metanorma-bsi", branch: "private-feature"
+gem "metanorma-nist", github: "metanorma/metanorma-nist", branch: "main"
+----
+
+The CI workflow uses `METANORMA_CI_PAT_TOKEN` for authentication to private repos.
+
+== Troubleshooting
+
+=== Bundle Install Fails
+
+**Symptom**: The CI job fails during `bundle install` with dependency resolution errors.
+
+**Solutions**:
+
+1. Verify branch names are correct
+2. Check that the specified branches exist and are accessible
+3. Ensure gem versions are compatible (check each gem's dependencies)
+4. Try running `bundle install` locally with the same Gemfile.devel
+
+=== Gems Aren't Being Overridden
+
+**Symptom**: Tests run but appear to use released gem versions instead of your PR.
+
+**Solutions**:
+
+1. Add diagnostic `warn` statements to your gem PR to confirm loading
+2. Check the CI logs for the `bundle install` output to see which gems were installed
+3. Verify the Gemfile.devel syntax is correct
+4. Ensure the file is committed to your PR branch
+
+=== Tests Fail Unexpectedly
+
+**Symptom**: Sample compilation fails with errors that don't appear in the gem's own tests.
+
+**This is the desired behavior!** The purpose of this mechanism is to catch integration issues.
+
+**Next Steps**:
+
+1. Download the error logs artifact from the GitHub Actions run
+2. Identify which sample(s) are failing
+3. Check if the failure is due to:
+   - An incompatible change in your PR (fix the PR)
+   - A bug in the sample document (fix the sample)
+   - A known limitation (document it)
+4. Consider whether your change needs coordination with other gem updates
+
+=== Private Repository Access
+
+**Symptom**: Can't access private sample repositories or gems.
+
+**Solutions**:
+
+1. Ensure you have the appropriate GitHub permissions
+2. For local testing, set up GitHub CLI authentication: `gh auth login`
+3. For CI, the workflow uses the `METANORMA_CI_PAT_TOKEN` secret automatically
+
+== Best Practices
+
+=== 1. Test Before Releasing Infrastructure Changes
+
+Always create a Gemfile.devel test before releasing changes to:
+
+* `isodoc`
+* `metanorma-standoc`
+* `metanorma-generic`
+* `metanorma-iso`
+
+These gems affect many downstream dependencies.
+
+=== 2. Test the Full Chain for Related PRs
+
+If you have PRs across multiple gems that work together, test them all simultaneously using Gemfile.devel before merging any of them.
+
+=== 3. Add Diagnostic Output During Testing
+
+Temporarily add `warn` statements to confirm gem loading:
+
+[source,ruby]
+----
+warn "DEBUG: isodoc #{Isodoc::VERSION} from PR #123"
+----
+
+Remove these before final merge.
+
+=== 4. Keep Gemfile.devel Out of Main Branch
+
+The `Gemfile.devel` file should only exist in feature branches for testing. Never commit it to `main` or `master`. (It's git-ignored, but be careful if you override this.)
+
+=== 5. Document Breaking Changes
+
+If your test reveals that a change will break compatibility, document this clearly in:
+
+* The PR description
+* The gem's CHANGELOG
+* Any related issue discussions
+
+=== 6. Coordinate Release Timing
+
+When multiple PRs need to be released together:
+
+1. Test all PRs together with Gemfile.devel
+2. Merge all PRs to their respective main branches
+3. Release gems in dependency order (base to top layer)
+4. Update dependent gems immediately after base releases
+
+== Git Configuration
+
+The `.gitignore` file includes:
+
+----
+/Gemfile.devel
+----
+
+This prevents accidental commits of `Gemfile.devel` to the main branch. The `.example` file is tracked to serve as documentation and a template.
+
+== Examples from Real Scenarios
+
+=== Example 1: svg_conform Integration Testing
+
+When `svg_conform` was added to the Metanorma stack, it should have been tested with Gemfile.devel:
+
+[source,ruby]
+----
+# Gemfile.devel
+gem "isodoc", github: "metanorma/isodoc", branch: "feature/svg-conform"
+gem "metanorma-standoc", github: "metanorma/metanorma-standoc", branch: "feature/svg-conform"
+----
+
+This would have caught compatibility issues across all samples before release.
+
+=== Example 2: Fontist Performance Testing
+
+Before releasing a Fontist upgrade:
+
+[source,ruby]
+----
+# Gemfile.devel
+gem "isodoc", github: "metanorma/isodoc", branch: "main"
+gem "fontist", github: "fontist/fontist", branch: "performance-improvements"
+----
+
+Performance impacts could be measured across the full sample set.
+
+=== Example 3: BSI Private Font Testing
+
+Testing BSI-specific changes with private fonts:
+
+[source,ruby]
+----
+# Gemfile.devel
+gem "metanorma-iso", github: "metanorma/metanorma-iso", branch: "main"
+gem "metanorma-bsi", github: "metanorma/metanorma-bsi", branch: "fix/font-handling"
+----
+
+The CI automatically sets up private fonts for BSI samples.
+
+== See Also
+
+* link:https://github.com/metanorma/metanorma-cli/issues/370[Issue #370: Hotfix test job]
+* link:https://github.com/metanorma/ci[metanorma/ci repository] - Shared CI workflows
+* `Gemfile.devel.example` - Template with examples
+* `.github/workflows/rake.yml` - Local workflow configuration