bmad-module-skill-forge 0.2.0

package/src/workflows/test-skill/steps-c/step-01-init.md

@@ -0,0 +1,185 @@
+---
+name: 'step-01-init'
+description: 'Discover skill, load forge tier state, validate inputs, create test report'
+
+nextStepFile: './step-02-detect-mode.md'
+outputFile: '{output_folder}/test-report-{skill_name}.md'
+templateFile: '../templates/test-report-template.md'
+sidecarFile: '{sidecar_path}/forge-tier.yaml'
+skillsOutputFolder: '{skills_output_folder}'
+---
+
+# Step 1: Initialize Test
+
+## STEP GOAL:
+
+Discover and validate the target skill, load forge tier state to determine analysis depth, and create the test report document from template.
+
+## MANDATORY EXECUTION RULES (READ FIRST):
+
+### Universal Rules:
+
+- 🛑 NEVER proceed if required inputs are missing — halt with clear error
+- 📖 CRITICAL: Read the complete step file before taking any action
+- 🔄 CRITICAL: When loading next step, ensure entire file is read
+- ⚙️ TOOL/SUBPROCESS FALLBACK: If any instruction references a subprocess, subagent, or tool you do not have access to, you MUST still achieve the outcome in your main context thread
+- ✅ YOU MUST ALWAYS SPEAK OUTPUT In your Agent communication style with the config `{communication_language}`
+
+### Role Reinforcement:
+
+- ✅ You are a skill auditor operating in Ferris's Audit mode
+- ✅ If you already have been given a name, communication_style and identity, continue to use those while playing this new role
+- ✅ Methodical, precise, evidence-based — zero hallucination
+- ✅ Every finding must trace to actual artifacts
+
+### Step-Specific Rules:
+
+- 🎯 Focus only on input discovery and validation — do NOT begin analysis
+- 🚫 FORBIDDEN to start coverage or coherence checks in this step
+- 💬 Report what was found clearly and concisely
+- 📋 Validate all required files exist before proceeding
+
+## EXECUTION PROTOCOLS:
+
+- 🎯 Discover skill path from user input or batch list
+- 💾 Create output document from {templateFile}
+- 📖 Load forge tier state from {sidecarFile}
+- 🚫 FORBIDDEN to proceed if SKILL.md is missing
+
+## CONTEXT BOUNDARIES:
+
+- Available: User-provided skill path, SKF module config
+- Focus: Input validation and tier detection only
+- Limits: Do not analyze skill content — just confirm it exists and is loadable
+- Dependencies: setup-forge must have run (forge-tier.yaml must exist)
+
+## MANDATORY SEQUENCE
+
+**CRITICAL:** Follow this sequence exactly. Do not skip, reorder, or improvise unless user explicitly requests a change.
+
+### 1. Receive Skill Path
+
+If skill path was provided as workflow argument, use it directly.
+
+If no path provided, ask:
+
+"**Which skill would you like to test?**
+
+Provide the skill path or name. I'll search in `{skillsOutputFolder}`.
+
+**Path or name:**"
+
+### 2. Validate Skill Exists
+
+Check that the skill directory contains required files:
+
+**Required files:**
+- `{skillsOutputFolder}/{skill_name}/SKILL.md` — the skill documentation
+- `{skillsOutputFolder}/{skill_name}/metadata.json` — skill metadata
+
+**If SKILL.md missing:**
+"**Error: SKILL.md not found at `{skillsOutputFolder}/{skill_name}/SKILL.md`**
+
+This skill has not been created yet. Run the **create-skill** workflow first."
+
+HALT — do not proceed.
+
+**If metadata.json missing:**
+"**Warning:** metadata.json not found. Proceeding with limited metadata. Some checks may be skipped."
+
+### 3. Load Forge Tier State
+
+Read `{sidecarFile}` to determine available analysis depth.
+
+**If forge-tier.yaml exists:**
+- Read `detected_tier` value (Quick, Forge, or Deep)
+- Read tool availability flags (ast_grep, gh_cli, qmd)
+
+**If forge-tier.yaml missing:**
+"**Warning:** Forge tier not detected. Defaulting to **Quick** tier (structural checks only).
+
+Run **setup-forge** to enable deeper analysis with AST-backed verification."
+
+Default to Quick tier and continue.
+
+### 4. Load Skill Metadata
+
+Read `metadata.json` to extract:
+- `skill_name` — display name
+- `skill_type` — individual or stack (needed for mode detection)
+- `source_path` — path to source code (if present)
+- `created_date` — when skill was created
+- `tier` — tier used during creation
+
+If source path override was provided as optional input, use that instead.
+
+### 5. Create Output Document
+
+Create `{outputFile}` from `{templateFile}` with initial frontmatter:
+
+```yaml
+---
+workflowType: 'test-skill'
+skillName: '{skill_name}'
+testMode: ''
+forgeTier: '{detected_tier}'
+testResult: ''
+score: ''
+threshold: ''
+testDate: '{current_date}'
+stepsCompleted: ['step-01-init']
+nextWorkflow: ''
+---
+```
+
+### 6. Report Initialization Status
+
+"**Test initialization complete.**
+
+**Skill:** {skill_name}
+**Path:** {skill_path}
+**Type:** {skill_type}
+**Forge Tier:** {detected_tier}
+**Source:** {source_path}
+
+**Proceeding to mode detection...**"
+
+### 7. Auto-Proceed
+
+Display: "**Proceeding to mode detection...**"
+
+#### Menu Handling Logic:
+
+- After initialization is complete, update {outputFile} frontmatter stepsCompleted, then immediately load, read entire file, then execute {nextStepFile}
+
+#### EXECUTION RULES:
+
+- This is an auto-proceed initialization step with no user choices
+- Proceed directly to next step after setup
+
+## CRITICAL STEP COMPLETION NOTE
+
+ONLY WHEN all required files are validated, forge tier is loaded, output document is created, and initialization status is reported, will you then load and read fully `{nextStepFile}` to execute mode detection.
+
+---
+
+## 🚨 SYSTEM SUCCESS/FAILURE METRICS
+
+### ✅ SUCCESS:
+
+- Skill path validated (SKILL.md exists)
+- Forge tier state loaded (or gracefully defaulted to Quick)
+- Skill metadata extracted
+- Output document created from template with initial frontmatter
+- Initialization status reported clearly
+- Auto-proceeded to step 02
+
+### ❌ SYSTEM FAILURE:
+
+- Proceeding without validating SKILL.md exists
+- Not loading forge tier state
+- Not creating output document before proceeding
+- Starting analysis in this step (coverage/coherence belongs to later steps)
+- Hardcoding paths instead of using frontmatter variables
+
+**Master Rule:** Skipping steps, optimizing sequences, or not following exact instructions is FORBIDDEN and constitutes SYSTEM FAILURE.

package/src/workflows/test-skill/steps-c/step-02-detect-mode.md

@@ -0,0 +1,162 @@
+---
+name: 'step-02-detect-mode'
+description: 'Determine naive (individual) vs contextual (stack) testing mode'
+
+nextStepFile: './step-03-coverage-check.md'
+outputFile: '{output_folder}/test-report-{skill_name}.md'
+---
+
+# Step 2: Detect Mode
+
+## STEP GOAL:
+
+Examine the skill metadata to determine whether this is an individual skill (naive mode — API surface coverage only) or a stack skill (contextual mode — full coherence validation including cross-references and integration patterns).
+
+## MANDATORY EXECUTION RULES (READ FIRST):
+
+### Universal Rules:
+
+- 🛑 NEVER override mode detection with assumptions — use actual metadata
+- 📖 CRITICAL: Read the complete step file before taking any action
+- 🔄 CRITICAL: When loading next step, ensure entire file is read
+- ⚙️ TOOL/SUBPROCESS FALLBACK: If any instruction references a subprocess, subagent, or tool you do not have access to, you MUST still achieve the outcome in your main context thread
+- ✅ YOU MUST ALWAYS SPEAK OUTPUT In your Agent communication style with the config `{communication_language}`
+
+### Role Reinforcement:
+
+- ✅ You are a skill auditor in Ferris's Audit mode
+- ✅ If you already have been given a name, communication_style and identity, continue to use those while playing this new role
+- ✅ Mode detection is deterministic — based on metadata, not judgment
+- ✅ Report the detected mode clearly and explain what it means for analysis depth
+
+### Step-Specific Rules:
+
+- 🎯 Focus only on mode detection — do NOT begin coverage or coherence analysis
+- 🚫 FORBIDDEN to start analyzing exports or references in this step
+- 💬 Explain what each mode means for the upcoming analysis
+- 📋 Set testMode in output frontmatter before proceeding
+
+## EXECUTION PROTOCOLS:
+
+- 🎯 Read skill metadata to determine type
+- 💾 Update {outputFile} frontmatter with testMode and append Test Summary section
+- 📖 Update stepsCompleted in {outputFile}
+- 🚫 FORBIDDEN to proceed without setting testMode
+
+## CONTEXT BOUNDARIES:
+
+- Available: Skill metadata loaded in step 01, forge tier state, output document
+- Focus: Mode detection only — classify skill type, set test mode
+- Limits: Do not analyze content — just determine the correct testing mode
+- Dependencies: step-01 must have validated skill exists and loaded metadata
+
+## MANDATORY SEQUENCE
+
+**CRITICAL:** Follow this sequence exactly. Do not skip, reorder, or improvise unless user explicitly requests a change.
+
+### 1. Examine Skill Type
+
+Read the skill metadata (loaded in step 01) and check:
+
+**Individual Skill indicators:**
+- `skill_type: 'individual'` in metadata
+- Single source library/package
+- No cross-references to other skills
+- Self-contained API surface
+
+**Stack Skill indicators:**
+- `skill_type: 'stack'` in metadata
+- References multiple skills or packages
+- Contains cross-references in SKILL.md
+- Integration patterns documented between components
+
+### 2. Determine Test Mode
+
+**IF individual skill → Naive Mode:**
+- Coverage check: documented exports vs source API surface
+- Coherence check: basic structural validation only (no cross-references to verify)
+- Scoring: redistributed weights (no coherence category)
+
+**IF stack skill → Contextual Mode:**
+- Coverage check: documented exports vs source API surface (same as naive)
+- Coherence check: full validation — cross-references exist, types match, integration patterns complete
+- Scoring: full category weights including coherence
+
+**IF metadata unclear or skill_type not set:**
+- Default to **naive mode** (conservative — fewer checks, less chance of false negatives from missing context)
+- Note the default in the report
+
+### 3. Update Output Document
+
+Update `{outputFile}` frontmatter:
+- Set `testMode: '{naive|contextual}'`
+
+Append the **Test Summary** section to `{outputFile}`:
+
+```markdown
+## Test Summary
+
+**Skill:** {skill_name}
+**Test Mode:** {naive|contextual}
+**Forge Tier:** {detected_tier}
+**Threshold:** {threshold}%
+
+**Mode Rationale:** {brief explanation of why this mode was selected}
+
+**Analysis Plan:**
+- Coverage Check: {what will be checked based on mode + tier}
+- Coherence Check: {what will be checked based on mode + tier}
+```
+
+### 4. Report Mode Detection
+
+"**Mode detected: {NAIVE|CONTEXTUAL}**
+
+**{skill_name}** is {an individual skill / a stack skill}, so we'll run in **{naive/contextual}** mode.
+
+**What this means:**
+- Coverage: {brief description of coverage scope}
+- Coherence: {brief description of coherence scope}
+- Scoring: {which weight distribution applies}
+
+**Proceeding to coverage check...**"
+
+### 5. Auto-Proceed
+
+Display: "**Proceeding to coverage check...**"
+
+#### Menu Handling Logic:
+
+- After mode detection is complete, update {outputFile} frontmatter stepsCompleted, then immediately load, read entire file, then execute {nextStepFile}
+
+#### EXECUTION RULES:
+
+- This is an auto-proceed detection step with no user choices
+- Proceed directly to next step after mode is set
+
+## CRITICAL STEP COMPLETION NOTE
+
+ONLY WHEN testMode is set in the output frontmatter and the Test Summary section is appended to the output document, will you then load and read fully `{nextStepFile}` to execute coverage check.
+
+---
+
+## 🚨 SYSTEM SUCCESS/FAILURE METRICS
+
+### ✅ SUCCESS:
+
+- Skill type correctly identified from metadata
+- Test mode set (naive or contextual) in output frontmatter
+- Test Summary section appended to output document
+- Mode rationale documented
+- Analysis plan stated based on mode + tier combination
+- Auto-proceeded to step 03
+
+### ❌ SYSTEM FAILURE:
+
+- Guessing mode without reading metadata
+- Not setting testMode in output frontmatter
+- Not appending Test Summary section
+- Starting analysis in this step
+- Hardcoding paths instead of using frontmatter variables
+
+**Master Rule:** Skipping steps, optimizing sequences, or not following exact instructions is FORBIDDEN and constitutes SYSTEM FAILURE.

package/src/workflows/test-skill/steps-c/step-03-coverage-check.md

@@ -0,0 +1,222 @@
+---
+name: 'step-03-coverage-check'
+description: 'Compare documented exports in SKILL.md against actual source API surface'
+
+nextStepFile: './step-04-coherence-check.md'
+outputFile: '{output_folder}/test-report-{skill_name}.md'
+scoringRulesFile: '../data/scoring-rules.md'
+---
+
+# Step 3: Coverage Check
+
+## STEP GOAL:
+
+Compare the exports, functions, classes, types, and interfaces documented in SKILL.md against the actual source code API surface. Identify missing documentation, undocumented exports, and signature mismatches. Analysis depth scales with forge tier.
+
+## MANDATORY EXECUTION RULES (READ FIRST):
+
+### Universal Rules:
+
+- 🛑 NEVER fabricate findings — every coverage result must trace to actual source code or SKILL.md content
+- 📖 CRITICAL: Read the complete step file before taking any action
+- 🔄 CRITICAL: When loading next step, ensure entire file is read
+- ⚙️ TOOL/SUBPROCESS FALLBACK: If any instruction references a subprocess, subagent, or tool you do not have access to, you MUST still achieve the outcome in your main context thread
+- ✅ YOU MUST ALWAYS SPEAK OUTPUT In your Agent communication style with the config `{communication_language}`
+
+### Role Reinforcement:
+
+- ✅ You are a skill auditor in Ferris's Audit mode — zero hallucination
+- ✅ If you already have been given a name, communication_style and identity, continue to use those while playing this new role
+- ✅ Every finding must include file:line citations from source code
+- ✅ Report what IS documented vs what SHOULD BE documented — facts only
+
+### Step-Specific Rules:
+
+- 🎯 Use subprocess optimization for per-file AST analysis (Pattern 2) when available
+- 💬 Subprocess returns structured findings only, not full file contents
+- 🚫 DO NOT BE LAZY — For EACH source file, launch a subprocess for deep analysis
+- ⚙️ If subprocess unavailable, perform analysis in main thread sequentially
+- 📋 Coverage depth must match the detected forge tier
+
+## EXECUTION PROTOCOLS:
+
+- 🎯 Load SKILL.md exports section and source files
+- 💾 Append Coverage Analysis section to {outputFile}
+- 📖 Update stepsCompleted in {outputFile}
+- 🚫 FORBIDDEN to proceed without completing all source file analysis
+
+## CONTEXT BOUNDARIES:
+
+- Available: SKILL.md, source files, forge tier, test mode from step 02
+- Focus: Export coverage comparison only — coherence is step 04
+- Limits: Do NOT validate cross-references or integration patterns (that's coherence)
+- Dependencies: step-02 must have set testMode and reported forge tier
+
+## MANDATORY SEQUENCE
+
+**CRITICAL:** Follow this sequence exactly. Do not skip, reorder, or improvise unless user explicitly requests a change.
+
+### 1. Extract Documented Exports from SKILL.md
+
+Read SKILL.md and extract all documented items:
+
+- **Functions:** name, parameters, return type, description
+- **Classes:** name, methods, properties
+- **Types/Interfaces:** name, fields
+- **Constants/Enums:** name, values
+- **Hooks/Patterns:** name, usage signature
+
+Build the **documented inventory** — a list of everything the SKILL.md claims the source provides.
+
+### 2. Analyze Source Code (Tier-Dependent)
+
+**Quick Tier (no tools):**
+- Read source files directly
+- Identify exported items by scanning for `export` keywords, `module.exports`, or language-specific export patterns
+- Compare against documented inventory by name matching
+- Cannot verify signatures — note as "unverified" in report
+
+**Forge Tier (ast-grep available):**
+DO NOT BE LAZY — For EACH source file in the skill's source path, launch a subprocess that:
+1. Uses ast-grep to extract all exported symbols with their full signatures
+2. Matches each export against the documented inventory
+3. Returns structured findings:
+
+```json
+{
+  "file": "src/utils.ts",
+  "exports_found": ["formatDate", "parseConfig", "ConfigType"],
+  "exports_documented": ["formatDate", "parseConfig"],
+  "missing_docs": ["ConfigType"],
+  "signature_mismatches": [
+    {
+      "name": "formatDate",
+      "source_sig": "(date: Date, format?: string) => string",
+      "documented_sig": "(date: Date) => string",
+      "issue": "missing optional parameter 'format'"
+    }
+  ]
+}
+```
+
+If subprocess unavailable, perform ast-grep analysis in main thread per file.
+
+**Deep Tier (ast-grep + gh + QMD):**
+- All Forge tier checks, plus:
+- Use gh CLI to verify source repository matches documented version
+- Cross-check type definitions against their source declarations
+- Verify re-exported symbols trace to their original source
+
+### 3. Build Coverage Results
+
+Aggregate findings across all source files:
+
+**Per-export status table:**
+
+| Export | Type | Documented | Signature Match | File:Line | Status |
+|--------|------|-----------|-----------------|-----------|--------|
+| {name} | function/class/type | yes/no | yes/no/unverified | src/file.ts:42 | PASS/FAIL/WARN |
+
+**Summary counts:**
+- Total exports in source: {N}
+- Documented in SKILL.md: {N}
+- Missing documentation: {N}
+- Signature mismatches: {N}
+- Undocumented in SKILL.md but not in source (stale docs): {N}
+
+### 4. Load Scoring Rules
+
+Load `{scoringRulesFile}` to determine category scores:
+
+- **Export Coverage:** (documented / total_exports) * 100
+- **Signature Accuracy:** (matching_signatures / total_documented) * 100 (Forge/Deep only, "N/A" for Quick)
+- **Type Coverage:** (documented_types / total_types) * 100 (Forge/Deep only, "N/A" for Quick)
+
+### 5. Append Coverage Analysis to Output
+
+Append the **Coverage Analysis** section to `{outputFile}`:
+
+```markdown
+## Coverage Analysis
+
+**Tier:** {forge_tier}
+**Source Path:** {source_path}
+**Files Analyzed:** {count}
+
+### Export Coverage
+
+| Export | Type | Documented | Signature | Source Location | Status |
+|--------|------|-----------|-----------|-----------------|--------|
+| ... per-export rows ... |
+
+### Coverage Summary
+
+- **Exports Found:** {N}
+- **Documented:** {N} ({percentage}%)
+- **Missing Documentation:** {N}
+- **Signature Mismatches:** {N}
+- **Stale Documentation:** {N}
+
+### Category Scores
+
+| Category | Score | Weight | Weighted |
+|----------|-------|--------|----------|
+| Export Coverage | {N}% | {weight}% | {weighted}% |
+| Signature Accuracy | {N}% | {weight}% | {weighted}% |
+| Type Coverage | {N}% | {weight}% | {weighted}% |
+```
+
+### 6. Report Coverage Results
+
+"**Coverage check complete.**
+
+**{skill_name}** — {forge_tier} tier analysis of {file_count} source files:
+
+- Exports: {documented}/{total} documented ({percentage}%)
+- Signatures: {matching}/{total} accurate ({percentage}% or N/A for Quick)
+- Types: {documented_types}/{total_types} covered ({percentage}% or N/A for Quick)
+
+**{N} issues found** — details in Coverage Analysis section.
+
+**Proceeding to coherence check...**"
+
+### 7. Auto-Proceed
+
+Display: "**Proceeding to coherence check...**"
+
+#### Menu Handling Logic:
+
+- After coverage analysis is complete, update {outputFile} frontmatter stepsCompleted, then immediately load, read entire file, then execute {nextStepFile}
+
+#### EXECUTION RULES:
+
+- This is an auto-proceed validation step with no user choices
+- Proceed directly to next step after coverage is analyzed
+
+## CRITICAL STEP COMPLETION NOTE
+
+ONLY WHEN all source files have been analyzed, the Coverage Analysis section has been appended to {outputFile}, and category scores have been calculated, will you then load and read fully `{nextStepFile}` to execute coherence check.
+
+---
+
+## 🚨 SYSTEM SUCCESS/FAILURE METRICS
+
+### ✅ SUCCESS:
+
+- All source files analyzed at appropriate tier depth
+- Every finding has file:line citation (Forge/Deep) or file-level reference (Quick)
+- Per-export status table complete
+- Category scores calculated per scoring rules
+- Coverage Analysis section appended to output document
+- Zero fabricated findings — all traceable to source
+
+### ❌ SYSTEM FAILURE:
+
+- Fabricating export names or signatures not in source code
+- Skipping source files (DO NOT BE LAZY)
+- Not scaling analysis depth to forge tier
+- Not calculating category scores
+- Reporting coverage without per-export evidence
+- Hardcoding paths instead of using frontmatter variables
+
+**Master Rule:** Skipping steps, optimizing sequences, or not following exact instructions is FORBIDDEN and constitutes SYSTEM FAILURE. Zero hallucination — every finding traces to code.