bmad-module-skill-forge 0.9.0 → 0.10.1-alpha.0

package/src/workflows/test-skill/steps-c/step-03-coverage-check.md

@@ -1,280 +0,0 @@
----
-name: 'step-03-coverage-check'
-description: 'Compare documented exports in SKILL.md against actual source API surface'
-
-nextStepFile: './step-04-coherence-check.md'
-outputFile: '{forge_data_folder}/{skill_name}/test-report-{skill_name}.md'
-scoringRulesFile: '../data/scoring-rules.md'
-sourceAccessProtocol: '../data/source-access-protocol.md'
----
-
-# Step 3: Coverage Check
-
-## STEP GOAL:
-
-Compare the exports, functions, classes, types, and interfaces documented in SKILL.md against the actual source code API surface. Identify missing documentation, undocumented exports, and signature mismatches. Analysis depth scales with forge tier.
-
-## MANDATORY EXECUTION RULES (READ FIRST):
-
-### Universal Rules:
-
-- 🛑 NEVER fabricate findings — every coverage result must trace to actual source code or SKILL.md content
-- 📖 CRITICAL: Read the complete step file before taking any action
-- 🔄 CRITICAL: When loading next step, ensure entire file is read
-- ⚙️ TOOL/SUBPROCESS FALLBACK: If any instruction references a subprocess, subagent, or tool you do not have access to, you MUST still achieve the outcome in your main context thread
-- ✅ YOU MUST ALWAYS SPEAK OUTPUT In your Agent communication style with the config `{communication_language}`
-
-### Role Reinforcement:
-
-- ✅ You are a skill auditor in Ferris's Audit mode — zero hallucination
-- ✅ If you already have been given a name, communication_style and identity, continue to use those while playing this new role
-- ✅ Every finding must include file:line citations from source code
-- ✅ Report what IS documented vs what SHOULD BE documented — facts only
-
-### Step-Specific Rules:
-
-- 🎯 Use subprocess optimization for per-file AST analysis when available
-- 💬 Subprocess returns structured findings only, not full file contents
-- 🚫 DO NOT BE LAZY — For EACH source file, launch a subprocess for deep analysis
-- ⚙️ If subprocess unavailable, perform analysis in main thread sequentially
-- 📋 Coverage depth must match the detected forge tier
-
-## EXECUTION PROTOCOLS:
-
-- 🎯 Load SKILL.md exports section and source files
-- 💾 Append Coverage Analysis section to {outputFile}
-- 📖 Update stepsCompleted in {outputFile}
-- 🚫 FORBIDDEN to proceed without completing all source file analysis
-
-## CONTEXT BOUNDARIES:
-
-- Available: SKILL.md, source files, forge tier, test mode from step 02
-- Focus: Export coverage comparison only — coherence is step 04
-- Limits: Do NOT validate cross-references or integration patterns (that's coherence)
-- Dependencies: step-02 must have set testMode and reported forge tier
-
-## MANDATORY SEQUENCE
-
-**CRITICAL:** Follow this sequence exactly. Do not skip, reorder, or improvise unless user explicitly requests a change.
-
-### 0. Check for Docs-Only Mode
-
-**If metadata.json indicates `confidence_tier: "Quick"` and all SKILL.md citations are `[EXT:...]` format (docs-only skill):**
-
-Coverage scoring adapts: instead of comparing SKILL.md against source code exports, compare SKILL.md documented items against themselves for internal completeness (every documented function has a description, parameters, and return type). Score based on documentation completeness rather than source coverage. Set `docs_only_mode: true` in context for step-05 scoring.
-
-"**Docs-only skill detected.** Coverage check evaluates documentation completeness rather than source code coverage."
-
-**If source-based skill:** Continue with standard coverage check below.
-
-### 0b. Load Source Access Protocol
-
-Load `{sourceAccessProtocol}` and follow both sections:
-1. **Source API Surface Definition** — determines what counts as the public API for coverage denominator
-2. **Source Access Resolution** — 5-state waterfall to determine how source files will be read and sets `analysis_confidence`
-
-### 1. Extract Documented Exports from SKILL.md
-
-Read SKILL.md and extract all documented items:
-
-- **Functions:** name, parameters, return type, description
-- **Classes:** name, methods, properties
-- **Types/Interfaces:** name, fields
-- **Constants/Enums:** name, values
-- **Hooks/Patterns:** name, usage signature
-
-Build the **documented inventory** — a list of everything the SKILL.md claims the source provides.
-
-**Split-body traversal:** If a `references/` directory exists alongside SKILL.md and SKILL.md's `## Full` headings are absent or stubs (not a stack skill's structural references), extend the documented inventory scan to include all `references/*.md` files. After split-body, Tier 2 content (Full API Reference, Full Type Definitions) lives in reference files — the inventory must reflect the full skill content regardless of where it resides.
-
-### 1b. Cross-Check Split-Body Consistency
-
-**Only execute if a `references/` directory exists alongside SKILL.md** (detected during split-body traversal in Section 1). Skip silently otherwise.
-
-For each function, class, type, or interface that appears in BOTH the SKILL.md body AND any `references/*.md` file, compare the documented signatures:
-
-- **Parameters:** name, type, order, optionality
-- **Return types:** exact type match
-- **Description:** no contradictions (brief vs detailed is acceptable; conflicting semantics is not)
-
-**SKILL.md body is authoritative.** When a mismatch is found, the reference file is the one that needs updating.
-
-Build a split-body consistency findings list:
-
-```json
-{
-  "cross_check_mismatches": [
-    {
-      "export": "formatDate",
-      "skill_md_line": 42,
-      "reference_file": "references/api-reference.md",
-      "reference_line": 18,
-      "issue": "SKILL.md shows (date: Date) => string, reference shows (date: Date, format?: string) => string"
-    }
-  ],
-  "exports_cross_checked": 12,
-  "mismatches_found": 1
-}
-```
-
-Flag each mismatch as **High severity** — signature inconsistency between SKILL.md body and reference files undermines agent trust. These findings feed into the gap report (step-06).
-
-### 2. Analyze Source Code (Tier-Dependent)
-
-Start from the package entry point (see 0b) and identify the public API surface. Then analyze those exports at the appropriate tier depth.
-
-**Quick Tier (no tools):**
-- Read the entry point file(s) directly
-- Identify public exports by scanning for `export` keywords, `module.exports`, `__init__.py` imports, or language-specific export patterns
-- Compare against documented inventory by name matching
-- Cannot verify signatures — note as "unverified" in report
-
-**Forge Tier (ast-grep available):**
-DO NOT BE LAZY — For EACH source file that defines public API exports, launch a subprocess that:
-1. Uses ast-grep to extract all exported symbols with their full signatures
-2. Matches each export against the documented inventory
-3. Returns structured findings:
-
-```json
-{
-  "file": "src/utils.ts",
-  "exports_found": ["formatDate", "parseConfig", "ConfigType"],
-  "exports_documented": ["formatDate", "parseConfig"],
-  "missing_docs": ["ConfigType"],
-  "signature_mismatches": [
-    {
-      "name": "formatDate",
-      "source_sig": "(date: Date, format?: string) => string",
-      "documented_sig": "(date: Date) => string",
-      "issue": "missing optional parameter 'format'"
-    }
-  ]
-}
-```
-
-If subprocess unavailable, perform ast-grep analysis in main thread per file.
-
-**Deep Tier (ast-grep + gh + QMD):**
-- All Forge tier checks, plus:
-- Use gh CLI to verify source repository matches documented version
-- Cross-check type definitions against their source declarations
-- Verify re-exported symbols trace to their original source
-
-### 3. Build Coverage Results
-
-Aggregate findings across all source files:
-
-**Per-export status table:**
-
-| Export | Type | Documented | Signature Match | File:Line | Status |
-|--------|------|-----------|-----------------|-----------|--------|
-| {name} | function/class/type | yes/no | yes/no/unverified | src/file.ts:42 | PASS/FAIL/WARN |
-
-**Summary counts:**
-- Total exports in source: {N}
-- Documented in SKILL.md: {N}
-- Missing documentation: {N}
-- Signature mismatches: {N}
-- Undocumented in SKILL.md but not in source (stale docs): {N}
-
-### 4. Load Scoring Rules
-
-Load `{scoringRulesFile}` to determine category scores:
-
-- **Export Coverage:** (documented / total_exports) * 100
-- **Signature Accuracy:** (matching_signatures / total_documented) * 100 (Forge/Deep only, "N/A" for Quick)
-- **Type Coverage:** (documented_types / total_types) * 100 (Forge/Deep only, "N/A" for Quick)
-
-**State 2 denominator validation:** When using provenance-map as the baseline (State 2), cross-reference the provenance-map entry count against `metadata.json`'s `exports[]` array before computing Export Coverage. If they diverge, use the union as the denominator per the source-access-protocol rules. Log the gap size if any.
-
-### 5. Append Coverage Analysis to Output
-
-Append the **Coverage Analysis** section to `{outputFile}`:
-
-```markdown
-## Coverage Analysis
-
-**Tier:** {forge_tier}
-**Source Access:** {analysis_confidence} (full | provenance-map | metadata-only | remote-only | docs-only)
-**Source Path:** {source_path}
-**Files Analyzed:** {count}
-
-### Export Coverage
-
-| Export | Type | Documented | Signature | Source Location | Status |
-|--------|------|-----------|-----------|-----------------|--------|
-| ... per-export rows ... |
-
-### Coverage Summary
-
-- **Exports Found:** {N}
-- **Documented:** {N} ({percentage}%)
-- **Missing Documentation:** {N}
-- **Signature Mismatches:** {N}
-- **Stale Documentation:** {N}
-
-### Category Scores
-
-| Category | Score |
-|----------|-------|
-| Export Coverage | {N}% |
-| Signature Accuracy | {N}% or N/A |
-| Type Coverage | {N}% or N/A |
-
-Note: Weight application is deferred to step-05 where all category weights are calculated after external validation availability is known.
-```
-
-### 6. Report Coverage Results
-
-"**Coverage check complete.**
-
-**{skill_name}** — {forge_tier} tier analysis of {file_count} source files:
-
-- Exports: {documented}/{total} documented ({percentage}%)
-- Signatures: {matching}/{total} accurate ({percentage}% or N/A for Quick)
-- Types: {documented_types}/{total_types} covered ({percentage}% or N/A for Quick)
-
-**{N} issues found** — details in Coverage Analysis section.
-
-**Proceeding to coherence check...**"
-
-### 7. Auto-Proceed
-
-Display: "**Proceeding to coherence check...**"
-
-#### Menu Handling Logic:
-
-- After coverage analysis is complete, update {outputFile} frontmatter stepsCompleted, then immediately load, read entire file, then execute {nextStepFile}
-
-#### EXECUTION RULES:
-
-- This is an auto-proceed validation step with no user choices
-- Proceed directly to next step after coverage is analyzed
-
-## CRITICAL STEP COMPLETION NOTE
-
-ONLY WHEN all source files have been analyzed, the Coverage Analysis section has been appended to {outputFile}, and category scores have been calculated, will you then load and read fully `{nextStepFile}` to execute coherence check.
-
----
-
-## 🚨 SYSTEM SUCCESS/FAILURE METRICS
-
-### ✅ SUCCESS:
-
-- All source files analyzed at appropriate tier depth; split-body references/ traversed when present
-- Every finding has file:line citation (Forge/Deep) or file-level reference (Quick)
-- Per-export status table complete
-- Category scores calculated per scoring rules
-- Coverage Analysis section appended to output document
-- Zero fabricated findings — all traceable to source
-
-### ❌ SYSTEM FAILURE:
-
-- Fabricating export names or signatures not in source code
-- Skipping source files (DO NOT BE LAZY)
-- Not scaling analysis depth to forge tier
-- Not calculating category scores
-- Reporting coverage without per-export evidence
-- Hardcoding paths instead of using frontmatter variables
-
-**Master Rule:** Skipping steps, optimizing sequences, or not following exact instructions is FORBIDDEN and constitutes SYSTEM FAILURE. Zero hallucination — every finding traces to code.

package/src/workflows/test-skill/steps-c/step-04-coherence-check.md

@@ -1,271 +0,0 @@
----
-name: 'step-04-coherence-check'
-description: 'Validate internal consistency — references, types, and integration patterns'
-
-nextStepFile: './step-04b-external-validators.md'
-outputFile: '{forge_data_folder}/{skill_name}/test-report-{skill_name}.md'
-outputFormatsFile: '../data/output-section-formats.md'
----
-
-# Step 4: Coherence Check
-
-## STEP GOAL:
-
-Validate internal consistency of the skill documentation. In contextual mode (stack skills): verify that all cross-references in SKILL.md point to real files, types match their declarations, and integration patterns are complete. In naive mode (individual skills): perform basic structural validation only.
-
-## MANDATORY EXECUTION RULES (READ FIRST):
-
-### Universal Rules:
-
-- 🛑 NEVER fabricate references or claim files exist without verifying — zero hallucination
-- 📖 CRITICAL: Read the complete step file before taking any action
-- 🔄 CRITICAL: When loading next step, ensure entire file is read
-- ⚙️ TOOL/SUBPROCESS FALLBACK: If any instruction references a subprocess, subagent, or tool you do not have access to, you MUST still achieve the outcome in your main context thread
-- ✅ YOU MUST ALWAYS SPEAK OUTPUT In your Agent communication style with the config `{communication_language}`
-
-### Role Reinforcement:
-
-- ✅ You are a skill auditor in Ferris's Audit mode — zero hallucination
-- ✅ If you already have been given a name, communication_style and identity, continue to use those while playing this new role
-- ✅ Every broken reference must cite exactly what was referenced and where it was expected
-- ✅ Report structural facts — do not interpret or editorialize
-
-### Step-Specific Rules:
-
-- 🎯 Use subprocess optimization: grep for references across files, then per-reference deep validation
-- 💬 Subprocess returns structured findings only, not full file contents
-- 🚫 DO NOT BE LAZY — For EACH reference in contextual mode, launch a subprocess to validate the target
-- ⚙️ If subprocess unavailable, perform validation in main thread sequentially
-- 📋 Analysis depth is conditional on testMode (naive vs contextual)
-
-## EXECUTION PROTOCOLS:
-
-- 🎯 Check testMode from output frontmatter to determine analysis depth
-- 💾 Append Coherence Analysis section to {outputFile}
-- 📖 Update stepsCompleted in {outputFile}
-- 🚫 FORBIDDEN to skip coherence check entirely — even naive mode runs basic structural validation
-
-## CONTEXT BOUNDARIES:
-
-- Available: SKILL.md, source files, testMode, forge tier, coverage results from step 03, evidence-report.md from skill's forge-data directory (Deep tier only)
-- Focus: Internal consistency and reference validation only — coverage was step 03
-- Limits: Do NOT recalculate coverage scores — use results from step 03
-- Dependencies: step-03 must have completed coverage analysis
-
-## MANDATORY SEQUENCE
-
-**CRITICAL:** Follow this sequence exactly. Do not skip, reorder, or improvise unless user explicitly requests a change.
-
-### 1. Check Test Mode
-
-Read `testMode` from `{outputFile}` frontmatter.
-
-**IF naive mode → Execute Naive Coherence (Section 2)**
-**IF contextual mode → Execute Contextual Coherence (Sections 3-5)**
-
-### 2. Naive Mode: Basic Structural Validation
-
-Perform lightweight structural checks:
-
-**Document structure:**
-- SKILL.md has required top-level sections (description, exports, usage)
-- Section headers are properly formatted
-- Code examples have language annotations
-- No broken markdown (unclosed code blocks, malformed tables)
-- If `scripts/` or `assets/` directory exists alongside SKILL.md, a "Scripts & Assets" section (Section 7b) should be present
-
-**Internal consistency:**
-- Exports referenced in usage examples match exports listed in exports section
-- Type names used in examples match documented types
-- No self-contradictions (e.g., function described as async but shown sync in example)
-
-Build a simple structural findings list:
-
-```json
-{
-  "structural_issues": [
-    {"type": "missing_section", "detail": "No 'Usage' section found"},
-    {"type": "broken_example", "detail": "Line 42: references undeclared function 'getConfig'"}
-  ],
-  "issues_found": 2
-}
-```
-
-**After naive coherence → Execute Section 2b if gate conditions met, then skip to Section 6 (Append Results)**
-
-### 2b. Migration/Deprecation Verification (Mode-Independent)
-
-**Gate check:** Execute this section ONLY IF both conditions are met:
-1. Forge tier is **Deep** (tool-gated)
-2. `{forge_data_folder}/{skill_name}/evidence-report.md` exists (data-gated)
-
-If either condition fails, skip silently and proceed to Section 6.
-
-**This check runs regardless of naive/contextual mode.** T2-future annotations are a property of the source code and enrichment data, not the skill type.
-
-Check whether SKILL.md contains a "Migration & Deprecation Warnings" section (Section 4b). Then check the skill's `evidence-report.md` (at `{forge_data_folder}/{skill_name}/evidence-report.md`) for T2-future annotation counts.
-
-- **If T2-future annotations > 0 AND Section 4b is absent:** Flag as Medium severity gap: "Migration section missing — T2-future annotations exist but Section 4b is not present in SKILL.md Tier 1."
-- **If T2-future annotations = 0 AND Section 4b is present:** Flag as Medium severity gap: "Migration section unexpected — Section 4b is present but no T2-future annotations were produced."
-- **If evidence-report.md is unavailable:** Skip this check silently. Note: "Section 4b verification skipped — evidence-report.md not found."
-
-Add findings to the coherence analysis results.
-
-**After Section 2b (naive path) → Skip to Section 6 (Append Results)**
-
-### 3. Contextual Mode: Extract References
-
-Scan SKILL.md for all cross-references:
-
-**Reference types to extract:**
-- File path references (`./path/to/file.ts`, `../shared/types.ts`)
-- Skill references (`See SKILL.md for {other-skill}`, `Integrates with {package}`)
-- Type imports (`import { Type } from './module'`)
-- Integration pattern references (middleware chains, plugin hooks, shared state)
-- Script/asset references (`scripts/{file}`, `assets/{file}`) in SKILL.md body
-
-Launch a subprocess to grep/regex SKILL.md for reference patterns and return all found references with line numbers as structured JSON (`references_found[]` with line, type, target fields). If subprocess unavailable, scan in main thread.
-
-### 4. Contextual Mode: Validate Each Reference
-
-DO NOT BE LAZY — For EACH reference found, launch a subprocess that:
-
-1. Checks if the target exists (file exists, skill exists, type is declared)
-2. If target exists, validates the reference is accurate:
-   - File path references: file exists at specified path
-   - Type imports: type is actually exported from the referenced module
-   - Skill references: referenced skill exists in skills output folder
-   - Integration patterns: documented pattern matches actual implementation
-   - Script/asset references: verify the referenced file exists in the skill's `scripts/` or `assets/` directory
-3. Returns structured validation JSON per reference (reference, line, target_exists, type_match, signature_match, issues[])
-
-If subprocess unavailable, validate each reference in main thread.
-
-4. **Scripts/assets directory check:** If a `scripts/` or `assets/` directory exists alongside SKILL.md, verify that a "Scripts & Assets" section (Section 7b) is present in SKILL.md. This directory-level check applies in both modes (naive mode performs it in Section 2; contextual mode performs it here alongside per-reference validation). Flag absence as Medium severity gap per `{scoringRulesFile}`.
-
-### 5. Contextual Mode: Check Integration Pattern Completeness
-
-For stack skills, verify integration patterns are complete:
-
-- **All documented integration points have corresponding code examples**
-- **Shared types are consistently used across referenced components**
-- **Middleware/plugin chains show complete flow, not fragments**
-- **Event handlers reference valid event types**
-
-Build integration completeness findings:
-
-```json
-{
-  "patterns_documented": 5,
-  "patterns_complete": 4,
-  "incomplete_patterns": [
-    {
-      "pattern": "Auth middleware chain",
-      "issue": "Shows middleware registration but not the handler function signature",
-      "line": 95
-    }
-  ]
-}
-```
-
-**Zero integration patterns:** If no integration patterns are documented in SKILL.md (e.g., a contextual-mode skill that uses shared types but has no middleware chains, plugin hooks, or event flows): record `patterns_documented: 0`, `patterns_complete: 0`. The coherence score will use reference validity alone — see `{scoringRulesFile}` Coherence Score Aggregation: "If no integration patterns exist, combined coherence equals reference validity."
-
-### 5b. Migration/Deprecation Verification (Contextual Path)
-
-**This section shares logic with Section 2b.** If you are on the contextual mode path (Sections 3-5), execute the migration check here using the same rules as Section 2b:
-
-**Gate check:** Execute ONLY IF both conditions are met:
-1. Forge tier is **Deep** (tool-gated)
-2. `{forge_data_folder}/{skill_name}/evidence-report.md` exists (data-gated)
-
-If either condition fails, skip silently.
-
-Check whether SKILL.md contains a "Migration & Deprecation Warnings" section (Section 4b). Then check the skill's `evidence-report.md` for T2-future annotation counts.
-
-- **If T2-future annotations > 0 AND Section 4b is absent:** Flag as Medium severity gap: "Migration section missing — T2-future annotations exist but Section 4b is not present in SKILL.md Tier 1."
-- **If T2-future annotations = 0 AND Section 4b is present:** Flag as Medium severity gap: "Migration section unexpected — Section 4b is present but no T2-future annotations were produced."
-- **If evidence-report.md is unavailable:** Skip this check silently. Note: "Section 4b verification skipped — evidence-report.md not found."
-
-Add findings to the coherence analysis results.
-
-### 5c. Calculate Coherence Scores
-
-**Contextual mode only.** Calculate coherence percentages using the formulas defined in `{scoringRulesFile}` — Coherence Score Aggregation section:
-
-```
-reference_validity = (valid_references / total_references) * 100
-integration_completeness = (complete_patterns / total_patterns) * 100
-combined_coherence = (reference_validity * 0.6) + (integration_completeness * 0.4)
-```
-
-**Edge case:** If no integration patterns are documented (patterns_documented = 0), combined coherence equals reference validity alone. Do not divide by zero.
-
-These values fill the `{percentage}%` placeholders in the output template loaded in Section 6.
-
-### 6. Append Coherence Analysis to Output
-
-Load `{outputFormatsFile}` and use the appropriate Coherence Analysis section format (naive or contextual) to append findings to `{outputFile}`.
-
-### 7. Report Coherence Results
-
-**For Naive Mode:**
-"**Coherence check complete (naive mode).**
-
-Basic structural validation of **{skill_name}**:
-- {N} structural issues found
-- Coherence category not scored (weight redistributed to coverage)
-
-**Proceeding to external validation...**"
-
-**For Contextual Mode:**
-"**Coherence check complete (contextual mode).**
-
-Reference validation of **{skill_name}**:
-- References: {valid}/{total} valid ({percentage}%)
-- Integration patterns: {complete}/{total} complete ({percentage}%)
-- Combined coherence: {percentage}%
-
-**{N} issues found** — details in Coherence Analysis section.
-
-**Proceeding to external validation...**"
-
-### 8. Auto-Proceed
-
-Display: "**Proceeding to external validation...**"
-
-#### Menu Handling Logic:
-
-- After coherence analysis is complete, update {outputFile} frontmatter stepsCompleted, then immediately load, read entire file, then execute {nextStepFile}
-
-#### EXECUTION RULES:
-
-- This is an auto-proceed validation step with no user choices
-- Proceed directly to next step after coherence is analyzed
-
-## CRITICAL STEP COMPLETION NOTE
-
-ONLY WHEN coherence analysis is complete (naive structural or contextual full validation), the Coherence Analysis section has been appended to {outputFile}, and coherence scores (if contextual) have been calculated, will you then load and read fully `{nextStepFile}` to execute external validation.
-
----
-
-## 🚨 SYSTEM SUCCESS/FAILURE METRICS
-
-### ✅ SUCCESS:
-
-- Correct mode path executed (naive structural OR contextual full)
-- All references validated in contextual mode (DO NOT BE LAZY)
-- Every broken reference cites exact location and expected target
-- Coherence Analysis section appended to output document
-- Coherence scores calculated (contextual) or weight redistribution noted (naive)
-- Zero fabricated findings — all traceable to artifacts
-
-### ❌ SYSTEM FAILURE:
-
-- Skipping coherence check entirely (even naive mode runs structural checks)
-- Fabricating reference validation results
-- Not checking every reference in contextual mode (DO NOT BE LAZY)
-- Running contextual checks on a naive-mode skill (unnecessary depth)
-- Running only naive checks on a contextual-mode skill (insufficient depth)
-- Hardcoding paths instead of using frontmatter variables
-
-**Master Rule:** Skipping steps, optimizing sequences, or not following exact instructions is FORBIDDEN and constitutes SYSTEM FAILURE. Zero hallucination — every finding traces to artifacts.