bmad-module-skill-forge 0.9.0 → 0.10.1-alpha.0

package/src/{workflows/test-skill → skf-test-skill}/steps-c/step-04b-external-validators.md

@@ -1,9 +1,6 @@
 ---
-name: 'step-04b-external-validators'
-description: 'Run external validation tools (skill-check, tessl) and capture scores for scoring step'
-
 nextStepFile: './step-05-score.md'
-outputFile: '{forge_data_folder}/{skill_name}/test-report-{skill_name}.md'
+outputFile: '{forge_version}/test-report-{skill_name}-{run_id}.md'
 ---
 
 # Step 4b: External Validators
@@ -12,41 +9,11 @@ outputFile: '{forge_data_folder}/{skill_name}/test-report-{skill_name}.md'
 
 Run external validation tools (`skill-check` and `tessl`) against the skill directory, capture their scores and findings, and append results to the test report. These tools catch complementary issues that internal coverage and coherence checks miss: `skill-check` validates spec compliance while `tessl` evaluates content quality and actionability.
 
-## MANDATORY EXECUTION RULES (READ FIRST):
-
-### Universal Rules:
-
-- 📖 CRITICAL: Read the complete step file before taking any action
-- 🎯 ALWAYS follow the exact instructions in the step file
-- ⚙️ TOOL/SUBPROCESS FALLBACK: If any instruction references a tool you do not have access to, you MUST still achieve the outcome in your main context thread
-- ✅ YOU MUST ALWAYS SPEAK OUTPUT in your Agent communication style with the config `{communication_language}`
-
-### Role Reinforcement:
-
-- ✅ You are a skill auditor running external quality gates
-- ✅ Both tools are optional — unavailability is a skip, never a halt
-- ✅ Report scores factually — do not editorialize on the results
-
-### Step-Specific Rules:
+## Rules
 
-- 🎯 Run each tool independently — one tool's failure does not affect the other
-- 🚫 FORBIDDEN to halt the workflow if either tool is unavailable or fails
-- 🚫 FORBIDDEN to modify skill content — this step only measures and reports
-- ⚠️ Ignore non-zero exit codes if the tool produces parseable output
-
-## EXECUTION PROTOCOLS:
-
-- 🎯 Follow MANDATORY SEQUENCE exactly
-- 💾 Append External Validation section to {outputFile}
-- 📖 Update stepsCompleted in {outputFile}
-- 📖 Store scores in context for step-05 scoring
-
-## CONTEXT BOUNDARIES:
-
-- Available: skill directory path, testMode, forge tier from previous steps
-- Focus: External tool execution and score capture only
-- Limits: Do NOT modify skill content or recalculate internal scores
-- Dependencies: steps 03 and 04 must have completed
+- Run each tool independently — one tool's failure does not affect the other
+- Do not halt the workflow if either tool is unavailable or fails
+- Read-only — do not modify skill content
 
 ## MANDATORY SEQUENCE
 
@@ -73,6 +40,8 @@ Before running external validators, check if `{forge_data_folder}/{skill_name}/e
 
 **Primary (git-tracked):** Run `git log -1 --format=%cI -- {skillDir}/SKILL.md` to get the last commit date of SKILL.md. Compare against the evidence report's generation date (from its frontmatter or the `## Validation Results` timestamp). If SKILL.md's last commit is newer, results are stale.
 
+**Primary-cross (single-commit bundle detection):** The git-commit-timestamp comparison can return a false "fresh" when `update-skill` commits a regenerated SKILL.md alongside an unchanged `evidence-report.md` in the same commit — both files share the same `%cI` even though the cached validation results inside the evidence report were produced during an earlier run. To catch this, after the Primary check also compare `{skillDir}/metadata.json`'s `generation_date` field against the evidence report's internal `## Validation Results` timestamp (or its frontmatter `generated` field). If `metadata.json.generation_date` is strictly newer than the evidence report's internal validation timestamp, treat results as stale regardless of git commit parity — SKILL.md was regenerated after the cached validation ran, so the scores no longer reflect current content. If `metadata.json` is missing or has no `generation_date`, skip this cross-check and rely on the git comparison alone.
+
 **Secondary (uncommitted changes):** Run `git diff --name-only -- {skillDir}/SKILL.md`. If output is non-empty, SKILL.md has uncommitted changes — treat results as stale regardless of commit dates. Also check `git diff --cached --name-only -- {skillDir}/SKILL.md` for staged-but-uncommitted changes — if non-empty, SKILL.md has been staged since last commit, treat results as stale.
 
 If SKILL.md was modified after the evidence report was generated (e.g., after update-skill), the cached results are stale — skip auto-reuse and proceed to section 2 for a fresh run.
@@ -83,20 +52,27 @@ If no evidence report exists, it contains no validation section, or results are
 
 ### 2. Run skill-check
 
-**Check availability:**
+**Check availability (short probe — 15s timeout):**
 
 ```bash
-npx skill-check -h
+timeout 15s npx --no-install skill-check -h 2>/dev/null
 ```
 
-If unavailable, record `skill_check_score: N/A` and skip to section 3.
+Use `--no-install` so the probe never triggers a slow cold-cache download (npx
+would otherwise fetch the package before printing help). Wrap in `timeout 15s`
+so a hung probe cannot stall the workflow — consistent with the 120s cap used
+on the actual validator run below. If the probe exits non-zero OR the 15s
+timeout trips (exit code `124`), record `skill_check_score: N/A` and skip to
+section 3.
 
-**Run validation:**
+**Run validation (S2 — 120s timeout):**
 
 ```bash
-npx skill-check check {skillDir} --format json --no-security-scan
+timeout 120s npx skill-check check {skillDir} --format json --no-security-scan
 ```
 
+If the command exits non-zero AND the exit code is `124` (GNU timeout's signal for the 120s wall-clock expiring), record `skill_check_score: N/A` with reason `timeout-120s`, log a warning, and skip to section 3. Other non-zero exits fall through to the regular JSON-parse path per the note below.
+
 **Parse JSON output** to extract:
 - `qualityScore` — overall score (0-100)
 - `diagnostics[]` — any remaining issues
@@ -110,30 +86,42 @@ Store in context: `skill_check_score`, `skill_check_diagnostics`
 
 ### 3. Run tessl
 
-**Check availability:**
+**Check availability (short probe — 15s timeout):**
 
 ```bash
-npx -y tessl --version
+timeout 15s npx --no-install -y tessl --version 2>/dev/null
 ```
 
-If unavailable, record `tessl_score: N/A` and skip to section 4.
+Same rationale as the skill-check probe above: `--no-install` + `timeout 15s`
+prevent a cold-cache fetch from stalling the workflow. If the probe exits
+non-zero OR the 15s timeout trips (exit code `124`), record
+`tessl_score: N/A` and skip to section 4.
+
+**Run review (S2 — 120s timeout):**
 
-**Run review:**
+The §2 probe (`npx --no-install -y tessl --version`) already resolved tessl via the caller's npm cache or a locally-installed binary on `$PATH`. Invoke the same binary for the review — do not re-pin to a registry-published version.
 
 ```bash
-npx -y tessl skill review {skillDir}
+# Use the tessl binary the §2 probe just verified. `--no-install` keeps
+# the review execution path identical to the probe; no fresh registry
+# fetch needed.
+timeout 120s npx --no-install -y tessl skill review {skillDir}
 ```
 
+Timeout handling mirrors skill-check: exit `124` → `tessl_score: N/A` with reason `timeout-120s`. If the percentage regex (`/(Description|Content|Review Score):\s*(\d+)%/`) returns fewer than three matches, record `tessl_score: N/A` with reason `parse-failure` and include the first 200 chars of output in evidence-report for debugging.
+
+**Registry-404 branch:** if the invocation emits `npm error 404 Not Found` or the npx wrapper exits with a not-found condition, record `tessl_score: N/A` with reason `pin-not-on-registry` and continue. This branch exists because tessl has historically shipped under shifting scope/tag combinations; do not HALT the workflow on a missing registry entry.
+
 **Parse the output** to extract:
 - `description_score` — percentage (e.g., 100%)
 - `content_score` — percentage (e.g., 45%)
-- `average_score` — percentage (e.g., 73%)
+- `review_score` — percentage (e.g., 73%)
 - `validation_result` — PASSED/FAILED
 - `judge_suggestions[]` — list of improvement suggestions
 
-The tessl output is human-readable text, not JSON. Parse the percentage values from lines like "Description: 100%", "Content: 45%", "Average Score: 73%".
+The tessl output is human-readable text, not JSON. Parse the percentage values from lines like "Description: 100%", "Content: 45%", "Review Score: 73%".
 
-Store in context: `tessl_description_score`, `tessl_content_score`, `tessl_average_score`, `tessl_suggestions`
+Store in context: `tessl_description_score`, `tessl_content_score`, `tessl_review_score`, `tessl_suggestions`
 
 **If tessl content score < 70%:** Flag a warning:
 
@@ -146,10 +134,10 @@ Store in context: `tessl_description_score`, `tessl_content_score`, `tessl_avera
 **If both tools ran:**
 
 ```
-external_score = (skill_check_score + tessl_average_score) / 2
+external_score = (skill_check_score + tessl_review_score) / 2
 ```
 
-Note: `skill_check_score` is 0-100, `tessl_average_score` is 0-100%. Both are on the same scale.
+Note: `skill_check_score` is 0-100, `tessl_review_score` is 0-100%. Both are on the same scale.
 
 **If only one tool ran:** Use that tool's score as the external score.
 
@@ -174,7 +162,7 @@ Append to `{outputFile}`:
 - **Validation:** {PASSED/FAILED}
 - **Description Score:** {score}%
 - **Content Score:** {score}%
-- **Average Score:** {score}%
+- **Review Score:** {score}%
 - **Suggestions:**
 {bulleted list of judge suggestions}
 
@@ -213,26 +201,3 @@ Append to `{outputFile}`:
 
 ONLY WHEN external validation results are appended to {outputFile} and scores are stored in context will you proceed to load `{nextStepFile}` for scoring.
 
----
-
-## 🚨 SYSTEM SUCCESS/FAILURE METRICS
-
-### ✅ SUCCESS:
-
-- skill-check executed and quality score captured (or skipped with N/A if unavailable)
-- tessl executed and description/content/average scores captured (or skipped with N/A)
-- Content quality warning raised if tessl content < 70%
-- Combined external score calculated
-- External Validation section appended to output
-- Scores stored in context for step-05
-- Auto-proceeded to step-05
-
-### ❌ SYSTEM FAILURE:
-
-- Halting the workflow because a tool is unavailable or fails
-- Not recording N/A when a tool is skipped (step-05 needs to know)
-- Modifying skill content based on validation results
-- Not parsing tessl content score (the key insight this tool provides)
-- Fabricating scores not from actual tool output
-
-**Master Rule:** External validators inform — they do not block. Run what's available, capture the scores, report factually, and proceed.

package/src/skf-test-skill/steps-c/step-05-score.md

@@ -0,0 +1,266 @@
+---
+nextStepFile: './step-06-report.md'
+outputFile: '{forge_version}/test-report-{skill_name}-{run_id}.md'
+scoringRulesFile: 'references/scoring-rules.md'
+sourceAccessProtocol: 'references/source-access-protocol.md'
+scoringScript: 'scripts/compute-score.py'
+---
+
+# Step 5: Score
+
+## STEP GOAL:
+
+Calculate the overall completeness score by aggregating coverage, coherence, and external validation category scores with the appropriate weight distribution (naive or contextual), apply the pass/fail threshold, and determine the test result.
+
+## Rules
+
+- Focus only on score calculation — do not generate remediation suggestions (Step 06)
+- Score must be deterministic — same inputs always produce same output
+- Show the math: category scores, weights, weighted contributions, total
+
+## MANDATORY SEQUENCE
+
+**CRITICAL:** Follow this sequence exactly. Do not skip, reorder, or improvise unless user explicitly requests a change.
+
+### 1. Load Scoring Rules
+
+Load `{scoringRulesFile}` to get:
+- Category weights (naive vs contextual distribution)
+- Default threshold (80%)
+- Tier-dependent scoring adjustments
+- Any custom threshold override from workflow input
+
+**Docs-only mode check:** If the Coverage Analysis section in `{outputFile}` notes docs-only mode (set by step-03 for skills with all `[EXT:...]` citations and no local source), apply Quick-tier weight redistribution: Signature Accuracy and Type Coverage are not scored, their weights (22% + 14%) are redistributed proportionally to remaining active categories. Coverage score is based on documentation completeness rather than source coverage (as calculated by step-03).
+
+### 2. Read Category Scores from Output
+
+Read `{outputFile}` and extract the category scores calculated in previous steps:
+
+**From Coverage Analysis (step 03):**
+- Export Coverage: {percentage}%
+- Signature Accuracy: {percentage}% or N/A (Quick tier)
+- Type Coverage: {percentage}% or N/A (Quick tier)
+
+**From Coherence Analysis (step 04):**
+- Combined Coherence: {percentage}% (contextual mode only)
+- Or: not scored (naive mode — weight redistributed)
+
+**From External Validation (step 04b):**
+- External Validation Score: {percentage}% (combined skill-check + tessl average)
+- Or: N/A (if neither tool was available — weight redistributed to other categories)
+
+### 3. Apply Weight Distribution
+
+**Read testMode from {outputFile} frontmatter.**
+
+#### 2b. Apply S9 State 2 Undercount Deduction (pre-script)
+
+If `analysis_confidence == 'provenance-map'` (State 2) AND step-03 recorded a provenance vs metadata divergence > 5% (see §4b in step-03), apply a 10-point deduction to `exportCoverage` BEFORE building the scoring input:
+
+```
+exportCoverage_adjusted = max(0, exportCoverage - 10)
+```
+
+Record in the report: `scoring_notes: State 2 undercount risk acknowledged — 10% deduction applied to Export Coverage (raw: {N}%, adjusted: {M}%)`. Use the adjusted value as the `exportCoverage` field in §3a below. The deduction is deterministic and does not change category weights or active-category counting.
+
+#### 3a. Construct Scoring Input JSON
+
+Build a JSON object from the data gathered in steps 1-2:
+
+```json
+{
+  "mode": "{testMode: contextual or naive}",
+  "tier": "{forge_tier: Quick, Forge, Forge+, or Deep}",
+  "docsOnly": "{true if docs_only_mode detected in step 03, else false}",
+  "state2": "{true if analysis_confidence is provenance-map, else false}",
+  "stackSkill": "{true if metadata.json.skill_type == 'stack', else false}",
+  "scores": {
+    "exportCoverage": "{export_coverage_percentage}",
+    "signatureAccuracy": "{signature_accuracy_percentage or null if N/A}",
+    "typeCoverage": "{type_coverage_percentage or null if N/A}",
+    "coherence": "{combined_coherence_percentage or null if naive mode}",
+    "externalValidation": "{external_validation_score or null if N/A}"
+  },
+  "threshold": "{custom_threshold or omit for default 80}"
+}
+```
+
+**Important:** Score values must be numbers (not strings). Use `null` (not `"N/A"`) for categories that were not scored. Read `metadata.json.skill_type` from `{resolved_skill_package}/metadata.json`; if the value is `"stack"`, set `stackSkill: true` and pass `null` for `signatureAccuracy` and `typeCoverage` (the categories will be redistributed per `{scoringRulesFile}` Stack Skills rule).
+
+#### 3b. Run the Scoring Script
+
+```bash
+python3 {scoringScript} '<JSON>'
+```
+
+Where `{scoringScript}` is the path resolved from the frontmatter variable (relative to the skill root, i.e., the skf-test-skill/ directory).
+
+Parse the JSON output. The script returns:
+- `weights` — final redistributed weights per category
+- `weightedScores` — weighted contribution per category
+- `totalScore` — the overall completeness score
+- `threshold` — the threshold used
+- `result` — `"PASS"`, `"FAIL"`, or **`"INCONCLUSIVE"`** (minimum-evidence floor — see `{scoringRulesFile}`)
+- `activeCategories` — list of categories that were scored
+- `skippedCategories` — list of categories that were skipped
+- `skipReasons` — why each category was skipped
+- `weightSum` — sum of final weights (should be ~100)
+- `inconclusiveReasons` — only present when `result == "INCONCLUSIVE"`; explains which floor clause tripped
+
+Use these values for Section 4 (pass/fail/inconclusive) and Section 6 (output formatting). **Never override the script's `result` value** — floor enforcement is centralized in the script for determinism.
+
+#### 3c. Fallback (if script execution fails)
+
+If the script is unavailable or returns an error, fall back to manual calculation:
+
+1. Select the weight table from `{scoringRulesFile}` for the detected mode (naive or contextual)
+2. Determine skip conditions: Quick tier/docsOnly/state2/stackSkill skip Signature Accuracy + Type Coverage; naive mode coherence is already 0; null external validation means skip it
+3. For each skipped category, set its weight to 0
+4. Compute sum of active category weights
+5. For each active category: `new_weight = old_weight / sum_active * 100`
+6. `weighted_score = new_weight / 100 * category_score`
+7. `total = sum of all weighted_scores`
+
+Report: "**Note:** Scoring script unavailable — calculated manually per scoring-rules.md."
+
+### 3d. Apply Post-Score Caps (BEFORE PASS/FAIL determination)
+
+The scoring script returns `totalScore` and a preliminary `result`. Two caps may override the script's PASS into FAIL — applied in this order, and ONLY if the script did NOT return INCONCLUSIVE (floor always takes precedence):
+
+**Cap 1 — Tooling degraded (B3):** if `analysisConfidence == "degraded"` or `toolingStatus` indicates a missing helper (`python3-missing`, `frontmatter-validator-missing`), set `effective_score = min(totalScore, threshold - 1)` and record `scoring_notes: tooling degraded — capped below threshold until helper restored`.
+
+**Cap 2 — Docs-only without external validators (S4):** if `docsOnly == true` AND `externalValidation` score was null (neither skill-check nor tessl ran), set `effective_score = min(effective_score, threshold - 1)` and record `scoring_notes: docs-only without external validators — capped below threshold per S4`.
+
+If either cap fired and `result` was `PASS`, change it to `FAIL`. If `result` was already `INCONCLUSIVE`, leave it — the evidence-floor verdict is never overridden by a cap.
+
+### 4. Determine Result (PASS / FAIL / INCONCLUSIVE)
+
+The scoring script enforces the minimum-evidence floor BEFORE comparing score vs threshold. Three possible outcomes:
+
+```
+IF result == "INCONCLUSIVE" — minimum-evidence floor tripped; not PASS, not FAIL
+IF score >= threshold       → PASS
+IF score < threshold        → FAIL
+```
+
+**INCONCLUSIVE floor clauses** (see `{scoringRulesFile}`):
+- `active_categories < 2` (after all redistribution), OR
+- `tier == "Quick"` AND Export Coverage is the sole scoring contributor
+
+**Tooling-degraded cap (B3):** If the `analysisConfidence` in output frontmatter is `degraded` (python3 missing, frontmatter validator missing, or other degraded state flagged in step-01), the step MUST cap the score at `threshold - 1` BEFORE the PASS/FAIL comparison. This forces a deterministic FAIL until tooling is restored. Do NOT override an INCONCLUSIVE result with the cap — INCONCLUSIVE remains the verdict.
+
+### 5. Determine Next Workflow Recommendation
+
+Based on test result:
+
+**IF PASS:**
+- `nextWorkflow: 'export-skill'` — skill is ready for export
+- **M5 — drift override:** if workflow context carries
+  `allow_workspace_drift: true` (set in step-01 §5b when the user passed
+  `--allow-workspace-drift` AND the workspace HEAD did not match
+  `metadata.source_commit`), the PASS is a **conditional PASS**:
+  - Write `testResult: 'pass-with-drift'` to the output frontmatter instead of
+    bare `'pass'`. The result contract (§4c of step-06) mirrors the same
+    value.
+  - Override `nextWorkflow` to `'update-skill'` — **refuse to recommend
+    `export-skill`**. The drift override weakens the workflow's strongest
+    false-positive guard (we tested against HEAD, not the pinned source); a
+    PASS under drift is not trustworthy enough to promote to export without a
+    clean re-test against the pinned commit.
+  - Record `scoring_notes: workspace drift overridden — PASS is conditional; re-run against pinned commit before export`.
+
+**IF FAIL:**
+- `nextWorkflow: 'update-skill'` — skill needs remediation before export
+
+**IF INCONCLUSIVE:**
+- `nextWorkflow: 'manual-review'` — evidence base is insufficient to grade the skill automatically. The test report records `inconclusiveReasons` from the scoring script. Surface to the user — do not auto-recommend export or update.
+
+### 6. Append Completeness Score to Output
+
+Append the **Completeness Score** section to `{outputFile}`:
+
+```markdown
+## Completeness Score
+
+### Score Breakdown
+
+| Category | Score | Weight | Weighted |
+|----------|-------|--------|----------|
+| Export Coverage | {N}% | {W}% | {WS}% |
+| Signature Accuracy | {N}% | {W}% | {WS}% |
+| Type Coverage | {N}% | {W}% | {WS}% |
+| Coherence | {N}% | {W}% | {WS}% |
+| External Validation | {N}% | {W}% | {WS}% |
+| **Total** | | **100%** | **{total}%** |
+
+### Result
+
+**Score:** {total}%
+**Threshold:** {threshold}%
+**Result:** **{PASS|FAIL|INCONCLUSIVE}**
+{If INCONCLUSIVE:}
+**Inconclusive Reasons:**
+{bulleted list from script `inconclusiveReasons`}
+
+**Weight Distribution:** {naive (redistributed) | contextual (full)}
+**Tier Adjustment:** {none | Quick tier — signature and type coverage not scored}
+**External Validators:** {both available | skill-check only | tessl only | none — weight redistributed}
+**Analysis Confidence:** {full | provenance-map | metadata-only | remote-only | docs-only}
+```
+
+If `analysis_confidence` is not `full`, append a degradation notice. **The notice must be confidence-aware** — see the degradation notice rules in `{sourceAccessProtocol}`:
+
+```markdown
+### Access Degradation Notice
+
+**Resolved via:** {analysis_confidence} {confidence breakdown if provenance-map, e.g., "(T1 AST-verified at compilation time)" or "(12 T1, 3 T1-low)"}
+**Impact:** {describe limitation — e.g., "Signature checks limited to name-matching. Source file:line citations from provenance-map, not live AST." — or "Provenance data is at highest confidence; no limitation." for all-T1 provenance-map}
+**Recommendation:** {confidence-dependent — see {sourceAccessProtocol} degradation notice rules. Do NOT recommend local clone when provenance-map entries are already T1.}
+```
+
+### 7. Update Output Frontmatter
+
+Update `{outputFile}` frontmatter:
+- `testResult: '{pass|pass-with-drift|fail|inconclusive}'` (lowercase; mirrors script `result`, with `pass-with-drift` substituted for `pass` when `allow_workspace_drift` was set and drift was observed — see §5 M5)
+- `score: '{total}%'`
+- `threshold: '{threshold}%'`
+- `analysisConfidence: '{full|degraded|provenance-map|metadata-only|remote-only|docs-only}'`
+- `nextWorkflow: '{export-skill|update-skill|manual-review}'`
+- Append `'step-05-score'` to `stepsCompleted`
+
+### 8. Report Score
+
+"**Completeness score calculated.**
+
+**{skill_name}:** **{total}%** — **{PASS|FAIL}**
+
+| Category | Score | Weighted |
+|----------|-------|----------|
+| Export Coverage | {N}% | {WS}% |
+| Signature Accuracy | {N}% | {WS}% |
+| Type Coverage | {N}% | {WS}% |
+| Coherence | {N}% | {WS}% |
+| External Validation | {N}% | {WS}% |
+
+**Threshold:** {threshold}%
+**Recommendation:** {export-skill if pass | update-skill if fail}
+
+**Proceeding to gap report...**"
+
+### 9. Auto-Proceed
+
+Display: "**Proceeding to gap report...**"
+
+#### Menu Handling Logic:
+
+- After score is calculated and frontmatter updated, immediately load, read entire file, then execute {nextStepFile}
+
+#### EXECUTION RULES:
+
+- This is an auto-proceed scoring step with no user choices
+- Proceed directly to next step after score is determined
+
+## CRITICAL STEP COMPLETION NOTE
+
+ONLY WHEN the score is calculated, pass/fail is determined, the Completeness Score section is appended to {outputFile}, and frontmatter is updated with testResult and score, will you then load and read fully `{nextStepFile}` to execute gap report generation.
+