bmad-module-skill-forge 2.2.0 → 3.0.0

package/src/skf-update-skill/references/validate.md

@@ -1,124 +0,0 @@
----
-nextStepFile: 'write.md'
-# Resolve `{hashContentHelper}` to the first existing path; HALT if neither
-# candidate exists. Check B uses its `manual-verify` subcommand to verify the
-# merged (on-disk) SKILL.md against the byte-exact [MANUAL] inventory captured
-# in step 1 §5 — the deterministic replacement for the LLM byte-identity
-# eyeball, which could silently pass a subtly truncated block.
-hashContentProbeOrder:
-  - '{project-root}/_bmad/skf/shared/scripts/skf-hash-content.py'
-  - '{project-root}/src/shared/scripts/skf-hash-content.py'
----
-
-<!-- Config: communicate in {communication_language}. -->
-
-# Step 5: Validate
-
-## STEP GOAL:
-
-Validate the merged skill content against the agentskills.io specification, verify all [MANUAL] sections survived the merge intact, and check confidence tier consistency across all re-extracted content. This is an advisory validation — findings are warnings, not blockers.
-
-## Rules
-
-- Focus only on validation — do not fix issues (that's the user's choice)
-- Validation is read-only — do not modify merged content
-- Run the two live structural checks (B, then C) in order; Checks A, D, E, F are deferred to step 6
-
-## Steps
-
-### 1. Check Tool Availability and Validation Timing
-
-Run: `npx skill-check -h`
-
-- If succeeds: skill-check is available for Checks A, E, F below
-- If fails: Use manual fallback paths in those checks
-
-**Important:** Do not assume availability — empirical check required.
-
-**Validation timing note:** Step-04 section 6b has already written SKILL.md to disk. External-tool checks against written files (skill-check Checks A, E, F) still run in **step 6 section 7** to co-locate external-tool validation with post-write verification. Check D (Provenance Completeness) is deterministic but needs both `metadata.json` and `provenance-map.json` on disk — neither exists yet at this step — so it is deferred to **step 6 section 6a**, which runs it via the completeness helper against the freshly-written artifacts. Structural checks (B, C) run here against the merged content — content on disk is byte-identical to the in-context copy.
-
-### 2. Run Validation Checks
-
-Per the §1 timing note, only Check B (a single deterministic `manual-verify` script) and Check C (an in-prompt tier-label consistency check) run live at this step; Checks A, D, E, F are deferred to step 6. Run B then C in order — one script plus one in-prompt comparison have nothing to fan out.
-
-**Check A — Spec Compliance (deferred to post-write):**
-
-Skill-check requires written files on disk. This check is deferred to step 6 section 7. Perform manual structural check only: verify merged SKILL.md has required sections (exports, usage patterns, conventions), verify export entries have name/type/signature/file:line reference, flag missing sections.
-
-**Check B — [MANUAL] Section Integrity:**
-
-Run the deterministic [MANUAL]-integrity verifier against the step-1 §5 inventory — do not eyeball byte-identity (a subtly truncated block reads as intact to the eye):
-
-```bash
-uv run {hashContentHelper} manual-verify {skill_package}/SKILL.md \
-    --inventory {manual_inventory}
-```
-
-Read the verdict JSON `{"preserved":[...], "modified":[...], "missing":[...], "moved":[...], "ok":bool}` (the on-disk SKILL.md is byte-identical to the merged in-context copy per the timing note above):
-- `ok == true` → status PASS. `preserved` blocks are byte-identical in place; any `moved` blocks are byte-identical but relocated with their logical parent section (report as informational, not a finding).
-- `modified` non-empty → FAIL: these blocks' byte-exact interiors changed (the truncation a marker-count/eyeball check would miss). List each by name.
-- `missing` non-empty → FAIL: these blocks lost their markers entirely. List each by name.
-
-Populate the §3 `manual_integrity` record from this verdict (`sections_verified` = inventory count, `sections_intact` = `len(preserved) + len(moved)`). This check is advisory here (findings inform, do not block); write.md §1 enforces the same `ok` verdict as a HALT gate before any derived artifact is written.
-
-**Check C — Confidence Tier Consistency:**
-- Verify all re-extracted exports have confidence labels (T1/T1-low/T2)
-- Verify tier labels match forge tier: Quick=T1-low only, Forge=T1 (T1-low for degraded), Forge+=T1 (same as Forge, CCC improves coverage not confidence), Deep=T1+T2
-- Flag mismatched or missing tier labels
-
-**Check D — Provenance Completeness (deferred to post-write):**
-
-The provenance map does not exist on disk until step 6 §3 writes it, and completeness is a deterministic set-diff (metadata `exports[]` vs provenance `entries[].export_name`) plus file:line citation resolution — not something to eyeball. This check is deferred to **step 6 section 6a**, which runs `skf-verify-provenance-completeness.py` against the just-written `metadata.json` and `provenance-map.json` and reads back `missing[]` / `orphaned[]` / `stale[]` findings. Do not attempt the set comparison here — there is no provenance map to compare against yet. The `Provenance` row in §5's summary is populated from §6a's result.
-
-**Check E — Diff Comparison (via skill-check):**
-
-**If available** and previous skill version exists: `npx skill-check diff <original-skill-dir> <updated-skill-dir>`
-
-Shows diagnostic changes between original and updated skill. Record diff results as informational context.
-
-**If unavailable or no previous version:** Skip with note.
-
-**Check F — Security Scan:**
-
-**If available**, run: `npx skill-check check <skill-dir> --format json` (security scan enabled by default).
-
-Record security findings as advisory warnings — they do not block the update.
-
-**If unavailable:** Skip with note: "Security scan skipped — skill-check tool unavailable"
-
-### 3. Aggregate Validation Results
-
-Compile results from all checks:
-
-```
-Validation Results:
-  spec_compliance: {status: PASS|WARN|FAIL, findings: [{severity, description, location}]}
-  manual_integrity: {status, sections_verified, sections_intact, findings}
-  confidence_consistency: {status, exports_checked, findings}
-  provenance_completeness: {status, entries_checked, findings}
-  diff_comparison: {status: PASS|SKIP, new_issues, fixed_issues, unchanged}
-  security_scan: {status: PASS|WARN|SKIP, findings}
-  quality_score: [0-100]  # from skill-check, if available
-```
-
-### 5. Display Validation Summary
-
-"**Validation Results:**
-
-| Check | Status | Findings |
-|-------|--------|----------|
-| Spec Compliance | {PASS/WARN/FAIL} | {count} findings (quality score: {score}/100) |
-| [MANUAL] Integrity | {PASS/WARN/FAIL} | {count} findings |
-| Confidence Tiers | {PASS/WARN/FAIL} | {count} findings |
-| Provenance | {PASS/WARN/FAIL} | {count} findings |
-| Diff Comparison | {PASS/SKIP} | {new} new, {fixed} fixed |
-| Security Scan | {PASS/WARN/SKIP} | {count} findings |
-
-**Overall: {ALL_PASS / WARNINGS_FOUND / FAILURES_FOUND}**"
-
-**If findings exist:** List each with severity, description, and location. Add: "Validation is advisory. Findings do not block the update."
-
-### 6. Route to Next Step
-
-This step auto-proceeds — no user choices; validation is advisory and does not block, it only informs. Once all validation checks have completed and findings are displayed, display "**Proceeding to write updated files...**", then load, fully read, and execute `{nextStepFile}` to write the updated files.
-

package/tools/validate-docs-drift.js

@@ -1,233 +0,0 @@
-/**
- * Documentation Drift Validator
- *
- * Verifies that SKF docs agree with the canonical oh-my-skills output on
- * every version number, commit SHA, and library reference the docs cite.
- *
- * What it checks:
- * - The anchor `skf_version` matches `package.json.version` (self-reference invariant
- *   declared in pinned.yaml's header; enforced here at CI/pre-commit time AND bumped
- *   by release.yaml on every release)
- * - Every skill listed in `docs/_data/pinned.yaml` actually exists at the
- *   claimed version and commit in $OMS/skills/<name>/<version>/<name>/metadata.json
- * - The metadata.json `version` field matches the anchor `version`
- * - The metadata.json `source_commit` field matches the anchor `source_commit`
- * - Every `.md` file under `docs/` is grepped for `<library> v?<x.y.z>` strings;
- *   any that don't match the currently-pinned version for that library are flagged
- * - Known illustrative libraries (hono, drizzle-orm, ...) are whitelisted and
- *   allowed to carry any version number
- *
- * Usage:
- *   node tools/validate-docs-drift.js             # validate, exit 1 on drift
- *   OMS=/path/to/oh-my-skills node tools/validate-docs-drift.js
- *
- * Exit codes:
- *   0 — no drift detected
- *   1 — drift found; fix the docs or update docs/_data/pinned.yaml
- *   2 — infrastructure failure (anchors file missing, oh-my-skills path
- *       unreachable, yaml parse error)
- */
-
-const fs = require('node:fs');
-const path = require('node:path');
-const yaml = require('js-yaml');
-
-const SKF_ROOT = path.resolve(__dirname, '..');
-const DOCS_DIR = path.join(SKF_ROOT, 'docs');
-const ANCHORS_PATH = path.join(DOCS_DIR, '_data', 'pinned.yaml');
-const PACKAGE_JSON_PATH = path.join(SKF_ROOT, 'package.json');
-
-function loadAnchors() {
-  if (!fs.existsSync(ANCHORS_PATH)) {
-    console.error(`error: anchors file not found at ${ANCHORS_PATH}`);
-    process.exit(2);
-  }
-  try {
-    return yaml.load(fs.readFileSync(ANCHORS_PATH, 'utf8'));
-  } catch (error) {
-    console.error(`error: could not parse ${ANCHORS_PATH}: ${error.message}`);
-    process.exit(2);
-  }
-}
-
-function checkSkfVersionAgainstPackageJson(anchors) {
-  const errors = [];
-  if (!fs.existsSync(PACKAGE_JSON_PATH)) {
-    errors.push(`CRITICAL: package.json not found at ${PACKAGE_JSON_PATH}`);
-    return errors;
-  }
-  let pkg;
-  try {
-    pkg = JSON.parse(fs.readFileSync(PACKAGE_JSON_PATH, 'utf8'));
-  } catch (error) {
-    errors.push(`CRITICAL: could not parse ${PACKAGE_JSON_PATH}: ${error.message}`);
-    return errors;
-  }
-  const anchorVersion = anchors.skf_version;
-  const packageVersion = pkg.version;
-  if (!anchorVersion) {
-    errors.push(`MISSING_ANCHOR: pinned.yaml has no skf_version key`);
-    return errors;
-  }
-  if (!packageVersion) {
-    errors.push(`MISSING_PKG_VERSION: package.json has no version field`);
-    return errors;
-  }
-  if (anchorVersion !== packageVersion) {
-    errors.push(
-      `SKF_VERSION_DRIFT: pinned.yaml skf_version "${anchorVersion}" !== package.json version "${packageVersion}" ` +
-        `— bump pinned.yaml or check that release.yaml's "Update pinned.yaml skf_version" step fired`,
-    );
-  }
-  return errors;
-}
-
-function resolveOmsPath(anchors) {
-  if (process.env.OMS) {
-    return path.resolve(process.env.OMS);
-  }
-  return path.resolve(SKF_ROOT, anchors.oh_my_skills_path);
-}
-
-// Accept either a full SHA match or a short-SHA prefix match, per
-// src/skf-update-skill/references/re-extract.md:36 — the pinned commit
-// is often stored as an 8-char short hash. Both anchors and metadata are
-// lowercased before comparison; blank / null values never match.
-function commitsMatch(anchorCommit, metadataCommit) {
-  if (!anchorCommit || !metadataCommit) return false;
-  const a = String(anchorCommit).toLowerCase();
-  const b = String(metadataCommit).toLowerCase();
-  return a === b || a.startsWith(b) || b.startsWith(a);
-}
-
-function checkCanonicalFiles(anchors, omsPath) {
-  const errors = [];
-  if (!fs.existsSync(omsPath) || !fs.statSync(omsPath).isDirectory()) {
-    errors.push(
-      `CRITICAL: oh_my_skills_path does not resolve to a directory: ${omsPath}`,
-      `  hint: set OMS=/path/to/oh-my-skills or edit docs/_data/pinned.yaml`,
-    );
-    return errors;
-  }
-
-  for (const [skillName, spec] of Object.entries(anchors.skills)) {
-    const metadataPath = path.join(omsPath, 'skills', skillName, spec.version, skillName, 'metadata.json');
-
-    if (!fs.existsSync(metadataPath)) {
-      errors.push(`MISSING: ${skillName}@${spec.version} — expected ${metadataPath}`);
-      continue;
-    }
-
-    let metadata;
-    try {
-      metadata = JSON.parse(fs.readFileSync(metadataPath, 'utf8'));
-    } catch (error) {
-      errors.push(`UNPARSEABLE: ${metadataPath} — ${error.message}`);
-      continue;
-    }
-
-    if (metadata.version !== spec.version) {
-      errors.push(`VERSION_DRIFT: ${skillName} — anchors say ${spec.version}, ` + `metadata.json says ${metadata.version}`);
-    }
-
-    if (!commitsMatch(spec.source_commit, metadata.source_commit)) {
-      const anchorShort = (spec.source_commit || '').slice(0, 12);
-      const realShort = (metadata.source_commit || '').slice(0, 12);
-      errors.push(`COMMIT_DRIFT: ${skillName} — anchors say ${anchorShort}, ` + `metadata.json says ${realShort}`);
-    }
-
-    if (metadata.confidence_tier !== spec.confidence_tier) {
-      errors.push(`TIER_DRIFT: ${skillName} — anchors say ${spec.confidence_tier}, ` + `metadata.json says ${metadata.confidence_tier}`);
-    }
-
-    if (metadata.source_authority !== spec.source_authority) {
-      errors.push(
-        `AUTHORITY_DRIFT: ${skillName} — anchors say ${spec.source_authority}, ` + `metadata.json says ${metadata.source_authority}`,
-      );
-    }
-  }
-
-  return errors;
-}
-
-function getMarkdownFiles(dir) {
-  const files = [];
-  function walk(currentDir) {
-    const entries = fs.readdirSync(currentDir, { withFileTypes: true });
-    for (const entry of entries) {
-      if (entry.name.startsWith('_')) continue;
-      const fullPath = path.join(currentDir, entry.name);
-      if (entry.isDirectory()) {
-        walk(fullPath);
-      } else if (entry.isFile() && entry.name.endsWith('.md')) {
-        files.push(fullPath);
-      }
-    }
-  }
-  walk(dir);
-  return files;
-}
-
-function checkDocsForStaleVersions(anchors) {
-  const errors = [];
-
-  // Build map: library-name → current-version
-  const libVersions = Object.fromEntries(Object.values(anchors.skills).map((spec) => [spec.library, spec.version]));
-  const libs = Object.keys(libVersions);
-  if (libs.length === 0) return errors;
-
-  const illustrative = new Set(anchors.illustrative_libraries || []);
-
-  // Pattern: <library-name><whitespace or @>v?<semver>
-  // Word-boundary anchored to avoid partial matches inside other words.
-  const libPattern = String.raw`\b(` + libs.map((l) => l.replaceAll(/[.*+?^${}()|[\\]\\\\]/g, String.raw`\\$&`)).join('|') + ')';
-  const versionPattern = String.raw`[\s@]+v?(\d+\.\d+\.\d+)`;
-  const regex = new RegExp(libPattern + versionPattern, 'gi');
-
-  for (const mdFile of getMarkdownFiles(DOCS_DIR)) {
-    const content = fs.readFileSync(mdFile, 'utf8');
-    const lines = content.split('\n');
-    for (const [idx, line] of lines.entries()) {
-      for (const match of line.matchAll(regex)) {
-        const lib = match[1].toLowerCase();
-        if (illustrative.has(lib)) continue;
-        const found = match[2];
-        const expected = libVersions[lib];
-        if (expected && found !== expected) {
-          const relPath = path.relative(SKF_ROOT, mdFile);
-          errors.push(`STALE_VERSION: ${relPath}:${idx + 1} — ${lib} v${found} (expected v${expected})`);
-        }
-      }
-    }
-  }
-
-  return errors;
-}
-
-function main() {
-  const anchors = loadAnchors();
-  const omsPath = resolveOmsPath(anchors);
-
-  const errors = [
-    ...checkSkfVersionAgainstPackageJson(anchors),
-    ...checkCanonicalFiles(anchors, omsPath),
-    ...checkDocsForStaleVersions(anchors),
-  ];
-
-  if (errors.length > 0) {
-    console.error('DRIFT DETECTED:\n');
-    for (const err of errors) {
-      console.error(`  - ${err}`);
-    }
-    console.error(`\n${errors.length} drift finding(s). Fix the docs or update docs/_data/pinned.yaml.`);
-    console.error('\nAnchors file: docs/_data/pinned.yaml');
-    console.error(`Checked against: ${omsPath}`);
-    process.exit(1);
-  }
-
-  const skillCount = Object.keys(anchors.skills).length;
-  console.log(`OK: skf_version ${anchors.skf_version} matches package.json; ${skillCount} skills checked against ${omsPath}, no drift.`);
-  process.exit(0);
-}
-
-main();