bmad-module-skill-forge 1.8.0 → 1.9.0

package/src/skf-test-skill/references/score.md

@@ -20,13 +20,16 @@ Load `{scoringRulesFile}` to get:
 - Category weights (naive vs contextual distribution)
 - Tier-dependent scoring adjustments
 
-**Resolve the pass threshold (precedence: CLI > scalar > bundled fallback):**
+**Resolve the pass threshold (precedence: CLI > pipeline default > scalar > bundled fallback):**
 
-1. If the workflow received `--threshold=<N>` on invocation, use that integer as `effective_threshold` (CLI wins).
-2. Else if the resolved `{defaultThreshold}` workflow-context variable (from SKILL.md On Activation §3 — `workflow.default_threshold` scalar, default `80`) is set, use it as `effective_threshold`.
-3. Else fall back to `80` (the bundled default — this branch should be unreachable when SKILL.md resolution ran correctly, but keeps the step robust if customize.toml resolution failed silently).
+1. If the workflow received `--threshold=<N>` on invocation, use that integer as `effective_threshold` (CLI wins). Set `threshold_source` = `"CLI override ({N}%)"`.
+2. Else if `{pipeline_default_threshold}` is set in workflow context (resolved by init.md §1b from the per-pipeline threshold lookup table when `{pipeline_alias}` is present), use it as `effective_threshold`. Set `threshold_source` = `"pipeline default ({pipeline_alias} → {N}%)"`.
+3. Else if the resolved `{defaultThreshold}` workflow-context variable (from SKILL.md On Activation §3 — `workflow.default_threshold` scalar, default `80`) is set, use it as `effective_threshold`. Set `threshold_source` = `"workflow default ({N}%)"`.
+4. Else fall back to `80` (the bundled default — this branch should be unreachable when SKILL.md resolution ran correctly, but keeps the step robust if customize.toml resolution failed silently). Set `threshold_source` = `"bundled fallback (80%)"`.
 
-Pass `effective_threshold` into the scoring-input JSON's `threshold` field in §3a (the compute-score.py script already honors this field). The CLI flag and the scalar feed the same downstream field; the script does not need to know which layer supplied the value.
+Store `threshold_source` in workflow context for use in the score report section.
+
+Pass `effective_threshold` into the scoring-input JSON's `threshold` field in §3a (the compute-score.py script already honors this field). The CLI flag, pipeline default, and the scalar all feed the same downstream field; the script does not need to know which layer supplied the value.
 
 **Docs-only mode check:** If the Coverage Analysis section in `{outputFile}` notes docs-only mode (set by step 3 for skills with all `[EXT:...]` citations and no local source), apply Quick-tier weight redistribution: Signature Accuracy and Type Coverage are not scored, their weights (22% + 14%) are redistributed proportionally to remaining active categories. Coverage score is based on documentation completeness rather than source coverage (as calculated by step 3).
 
@@ -82,7 +85,7 @@ Build a JSON object from the data gathered in steps 1-2:
     "coherence": "{combined_coherence_percentage or null if naive mode}",
     "externalValidation": "{external_validation_score or null if N/A}"
   },
-  "threshold": "{effective_threshold from §1 — CLI --threshold wins, then workflow.default_threshold scalar, then 80}"
+  "threshold": "{effective_threshold from §1 — CLI --threshold wins, then pipeline default, then workflow.default_threshold scalar, then 80}"
 }
 ```
 
@@ -150,6 +153,85 @@ IF score < threshold        → FAIL
 
 **Tooling-degraded cap:** If the `analysisConfidence` in output frontmatter is `degraded` (python3 missing, frontmatter validator missing, or other degraded state flagged in step 1), the step MUST cap the score at `threshold - 1` BEFORE the PASS/FAIL comparison. This forces a deterministic FAIL until tooling is restored. Do NOT override an INCONCLUSIVE result with the cap — INCONCLUSIVE remains the verdict.
 
+### 4b. Threshold Fallback and Evidence Report
+
+After §4 determines the result but before §5 recommends the next workflow, check whether a threshold fallback applies. The fallback converts a FAIL into a PASS at the 80% floor when the score is between 80% and the effective threshold, and documents the quality compromise in an evidence report.
+
+**Fallback trigger conditions:**
+
+```
+IF result == "FAIL"
+  AND totalScore >= 80
+  AND effective_threshold > 80
+THEN
+  → THRESHOLD FALLBACK triggered
+```
+
+**Do NOT trigger fallback when:**
+- `result == "INCONCLUSIVE"` — the evidence-floor verdict is never overridden by fallback
+- `totalScore < 80` — that is a genuine FAIL (floor not met)
+- `effective_threshold == 80` — there is nothing to fall back from
+
+**When fallback triggers:**
+
+1. Record `threshold_fallback: true`, `original_threshold: {effective_threshold}`, `fallback_threshold: 80` in workflow context.
+2. Override `result` to `"PASS"`.
+3. Set `effective_threshold = 80` for use by §5/§6/§7/§8.
+4. Generate the evidence report (§4b.1 below).
+
+#### 4b.1 Generate Evidence Report
+
+Write the evidence report to `{forge_version}/evidence-report-fallback.md`. The report documents the quality compromise for audit purposes.
+
+**Read gap entries:** extract findings from the Coverage Analysis and Coherence Analysis sections in `{outputFile}` — list each gap with severity (Critical through Low).
+
+**Check for prior remediation:** glob `{forge_version}/test-report-{skill_name}-*.md` for a prior test report. If found, note the path — this implies remediation was attempted between runs. If not found, note "first test run — no prior remediation cycle".
+
+**Check for post-score cap:** if Cap 1 or Cap 2 (§3d) fired, note the cap reason in the remediation section: "Score capped due to {cap_reason} — address tooling to test at the higher threshold."
+
+**Evidence report template:**
+
+```markdown
+# Evidence Report: Threshold Fallback
+
+**Skill:** {skill_name}
+**Date:** {ISO-8601 timestamp}
+**Run ID:** {run_id}
+
+## Threshold Summary
+
+| Field | Value |
+|-------|-------|
+| Attempted Threshold | {original_threshold}% |
+| Achieved Score | {totalScore}% |
+| Threshold Source | {threshold_source} |
+| Final Accepted Threshold | 80% |
+
+## Findings Preventing Higher Threshold
+
+{For each gap entry from Coverage Analysis and Coherence Analysis:}
+- **{GAP-NNN}: {title}** — Severity: {severity}, Category: {category}
+
+{Count: N critical, M high, P medium, Q low, R info findings}
+
+## Remediation Context
+
+{If prior test report exists:}
+A prior test run was found at `{prior_report_path}`, indicating remediation was attempted between runs.
+
+{If no prior test report:}
+No prior test report found for this skill version — this is the first test run.
+
+{If post-score cap was active:}
+**Note:** Score was capped due to {cap_reason}. Address tooling limitations to test at the higher threshold.
+
+## Conclusion
+
+Skill accepted at 80% floor (original target: {original_threshold}%). The {N} findings above prevented meeting the higher threshold. Review and address findings before the next pipeline run to achieve the {original_threshold}% target.
+```
+
+Record `evidence_report_path: '{forge_version}/evidence-report-fallback.md'` in workflow context for use by §6/§7/§8 and by report.md.
+
 ### 5. Determine Next Workflow Recommendation
 
 Based on test result:
@@ -203,6 +285,9 @@ Append the **Completeness Score** section to `{outputFile}`:
 **Inconclusive Reasons:**
 {bulleted list from script `inconclusiveReasons`}
 
+**Threshold Source:** {threshold_source}
+{If threshold_fallback is true:}
+**Threshold Fallback:** scored {totalScore}% against {original_threshold}% target — accepted at 80% floor. Evidence report: {evidence_report_path}
 **Weight Distribution:** {naive (redistributed) | contextual (full)}
 **Tier Adjustment:** {none | Quick tier — signature and type coverage not scored}
 **External Validators:** {both available | skill-check only | tessl only | none — weight redistributed}
@@ -225,6 +310,8 @@ Update `{outputFile}` frontmatter:
 - `testResult: '{pass|pass-with-drift|fail|inconclusive}'` (lowercase; mirrors script `result`, with `pass-with-drift` substituted for `pass` when `allow_workspace_drift` was set and drift was observed — see §5 drift override)
 - `score: '{total}%'`
 - `threshold: '{threshold}%'`
+- `thresholdSource: '{threshold_source}'`
+- When `threshold_fallback` is true, add: `thresholdFallback: true`, `originalThreshold: '{original_threshold}%'`, `evidenceReportPath: '{evidence_report_path}'`
 - `analysisConfidence: '{full|degraded|provenance-map|metadata-only|remote-only|docs-only}'`
 - `nextWorkflow: '{export-skill|update-skill|manual-review}'`
 - Append `'score'` to `stepsCompleted`
@@ -244,6 +331,8 @@ Update `{outputFile}` frontmatter:
 | External Validation | {N}% | {WS}% |
 
 **Threshold:** {threshold}%
+{If threshold_fallback is true:}
+**Threshold fallback:** scored {totalScore}% against {original_threshold}% target — accepted at 80% floor. Evidence report: {evidence_report_path}
 **Recommendation:** {export-skill if pass | update-skill if fail}
 
 **Proceeding to gap report...**"

package/src/skf-test-skill/references/step-hard-gate.md

@@ -0,0 +1,73 @@
+---
+nextStepFile: 'score.md'
+outputFile: '{forge_version}/test-report-{skill_name}-{run_id}.md'
+---
+
+<!-- Config: communicate in {communication_language}. -->
+
+# Step 4c: Hard Gate
+
+## STEP GOAL:
+
+Scan accumulated findings from coverage and coherence analysis for critical or high severity gaps. If any exist, block the pipeline with a clear listing of every blocking finding — no coverage score is computed or reported. If only medium, low, or info findings exist, pass through to scoring.
+
+### §1. Read Findings from Output
+
+Read `{outputFile}` and scan the **Coverage Analysis** and **Coherence Analysis** sections for GAP entries. Each GAP entry contains a severity marker in this format:
+
+```
+**Severity:** {Critical|High|Medium|Low|Info}
+```
+
+Extract every line matching `**Severity:** Critical` or `**Severity:** High`. For each match, also capture the parent GAP heading (`### GAP-{NNN}: {title}`) and the `**Source:**` line to build a blocking-findings list.
+
+### §2. Evaluate Gate
+
+**Count blocking findings** (Critical + High severity).
+
+**IF blocking findings exist → BLOCK (§3)**
+
+**IF no blocking findings → PASS (§4)**
+
+### §3. Block — Critical/High Findings Detected
+
+The hard gate blocks the pipeline. No scoring step runs.
+
+Update `{outputFile}` frontmatter:
+- Set `testResult: 'fail'`
+- Append `'hard-gate'` to `stepsCompleted`
+
+Report to the user:
+
+"**Hard gate BLOCKED — {N} critical/high finding(s) must be resolved before scoring.**
+
+| # | GAP | Severity | Source |
+|---|-----|----------|--------|
+{for each blocking finding:}
+| {i} | {GAP-NNN}: {title} | {severity} | {source} |
+
+**{M} medium/low/info findings also noted (non-blocking).**
+
+**Action required:** resolve all Critical and High findings, then re-run test-skill.
+**Recommended next step:** update-skill"
+
+**Headless envelope (if `{headless_mode}`):** emit to **stderr**:
+
+```
+SKF_TEST_RESULT_JSON: {"status":"error","skill_name":"{skill_name}","verdict":"FAIL","score":null,"threshold":null,"report_path":"{outputFile}","next_workflow":"update-skill","exit_code":2,"halt_reason":"hard-gate-blocked"}
+```
+
+HALT — do not chain to `{nextStepFile}`.
+
+### §4. Pass — No Critical/High Findings
+
+The hard gate passes. Medium, low, and info findings are documented in the gap report but do not block.
+
+Update `{outputFile}` frontmatter:
+- Append `'hard-gate'` to `stepsCompleted`
+
+Report:
+
+"**Hard gate passed** — {N} medium/low/info finding(s) noted, proceeding to scoring."
+
+Update stepsCompleted, then load and execute `{nextStepFile}`.

package/src/skf-test-skill/templates/test-report-template.md

@@ -28,6 +28,7 @@ Anchor / Step mapping:
   Coverage Analysis  → coverage-check
   Coherence Analysis → coherence-check
   External Validation→ external-validators
+  (hard gate)        → step-hard-gate (reads findings, blocks or passes; no report section)
   Completeness Score → score
   Gap Report         → report (includes Discovery Quality subsection)
 -->