bmad-module-skill-forge 2.0.0 → 2.0.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/marketplace.json +1 -1
- package/.nvmrc +1 -1
- package/docs/_data/pinned.yaml +1 -1
- package/package.json +2 -2
- package/src/knowledge/skill-lifecycle.md +1 -1
- package/src/module-help.csv +2 -2
- package/src/shared/scripts/skf-comention-pairs.py +350 -0
- package/src/shared/scripts/skf-count-tokens.py +222 -0
- package/src/shared/scripts/skf-detect-docs.py +232 -1
- package/src/shared/scripts/skf-detect-language.py +77 -0
- package/src/shared/scripts/skf-detect-tools.py +105 -2
- package/src/shared/scripts/skf-enumerate-stack-skills.py +111 -2
- package/src/shared/scripts/skf-find-cycles.py +246 -0
- package/src/shared/scripts/skf-hash-content.py +487 -2
- package/src/shared/scripts/skf-manifest-ops.py +67 -1
- package/src/shared/scripts/skf-rebuild-managed-sections.py +180 -0
- package/src/shared/scripts/skf-recommend-scope-type.py +1 -2
- package/src/shared/scripts/skf-render-metadata-stats.py +499 -0
- package/src/shared/scripts/skf-rewrite-skill-name.py +308 -0
- package/src/shared/scripts/skf-shard-body.py +461 -0
- package/src/shared/scripts/skf-skill-inventory.py +139 -3
- package/src/shared/scripts/skf-structural-diff.py +307 -67
- package/src/shared/scripts/skf-validate-brief-schema.py +39 -1
- package/src/shared/scripts/skf-validate-feasibility-report.py +234 -0
- package/src/shared/scripts/skf-validate-frontmatter.py +17 -12
- package/src/shared/scripts/skf-validate-output.py +443 -10
- package/src/shared/scripts/skf-validate-pins.py +1 -1
- package/src/shared/scripts/skf-verify-no-trace.py +214 -0
- package/src/shared/scripts/skf-verify-provenance-completeness.py +439 -0
- package/src/skf-analyze-source/SKILL.md +17 -32
- package/src/skf-analyze-source/customize.toml +2 -0
- package/src/skf-analyze-source/references/auto-docs-only.md +116 -0
- package/src/skf-analyze-source/references/continue.md +8 -13
- package/src/skf-analyze-source/references/generate-briefs.md +28 -12
- package/src/skf-analyze-source/references/headless-contract.md +33 -0
- package/src/skf-analyze-source/references/health-check.md +1 -2
- package/src/skf-analyze-source/references/identify-units.md +25 -36
- package/src/skf-analyze-source/references/init.md +12 -25
- package/src/skf-analyze-source/references/map-and-detect.md +8 -20
- package/src/skf-analyze-source/references/recommend.md +4 -14
- package/src/skf-analyze-source/references/scan-project.md +3 -11
- package/src/skf-analyze-source/references/step-auto-scope.md +42 -180
- package/src/skf-analyze-source/references/unit-detection-heuristics.md +2 -2
- package/src/skf-audit-skill/SKILL.md +9 -6
- package/src/skf-audit-skill/customize.toml +2 -0
- package/src/skf-audit-skill/references/init.md +58 -32
- package/src/skf-audit-skill/references/re-index.md +15 -34
- package/src/skf-audit-skill/references/report.md +11 -58
- package/src/skf-audit-skill/references/semantic-diff.md +4 -22
- package/src/skf-audit-skill/references/severity-classify.md +53 -87
- package/src/skf-audit-skill/references/step-doc-drift.md +49 -19
- package/src/skf-audit-skill/references/structural-diff.md +43 -68
- package/src/skf-brief-skill/SKILL.md +7 -36
- package/src/skf-brief-skill/assets/scope-templates.md +1 -1
- package/src/skf-brief-skill/customize.toml +14 -0
- package/src/skf-brief-skill/references/analyze-target.md +6 -14
- package/src/skf-brief-skill/references/confirm-brief.md +2 -14
- package/src/skf-brief-skill/references/draft-checkpoint.md +8 -8
- package/src/skf-brief-skill/references/gather-intent.md +19 -22
- package/src/skf-brief-skill/references/headless-args.md +1 -1
- package/src/skf-brief-skill/references/health-check.md +1 -22
- package/src/skf-brief-skill/references/invocation-contract.md +37 -0
- package/src/skf-brief-skill/references/scope-definition.md +4 -14
- package/src/skf-brief-skill/references/step-auto-brief.md +1 -5
- package/src/skf-brief-skill/references/step-auto-validate.md +1 -5
- package/src/skf-brief-skill/references/version-resolution.md +1 -1
- package/src/skf-brief-skill/references/write-brief.md +17 -10
- package/src/skf-campaign/SKILL.md +15 -84
- package/src/skf-campaign/customize.toml +2 -0
- package/src/skf-campaign/manifest.yaml +4 -0
- package/src/skf-campaign/references/campaign-contracts.md +67 -0
- package/src/skf-campaign/references/campaign-directive-spec.md +14 -103
- package/src/skf-campaign/references/step-01-setup.md +4 -5
- package/src/skf-campaign/references/step-02-strategy.md +8 -3
- package/src/skf-campaign/references/step-03-pins.md +1 -1
- package/src/skf-campaign/references/step-04-provenance.md +1 -1
- package/src/skf-campaign/references/step-05-skill-loop.md +1 -1
- package/src/skf-campaign/references/step-06-batch.md +7 -6
- package/src/skf-campaign/references/step-11-maintenance.md +2 -2
- package/src/skf-campaign/references/step-resume.md +9 -8
- package/src/skf-campaign/scripts/campaign-deps.py +10 -1
- package/src/skf-campaign/scripts/campaign-render-batch.py +199 -0
- package/src/skf-campaign/scripts/campaign-status.py +162 -0
- package/src/skf-campaign/templates/campaign-brief-template.yaml +1 -1
- package/src/skf-create-skill/SKILL.md +14 -3
- package/src/skf-create-skill/assets/compile-assembly-rules.md +17 -38
- package/src/skf-create-skill/assets/skill-sections.md +9 -100
- package/src/skf-create-skill/assets/tessl-dismissal-rules.md +4 -13
- package/src/skf-create-skill/customize.toml +13 -0
- package/src/skf-create-skill/references/authoritative-files-protocol.md +7 -7
- package/src/skf-create-skill/references/compile.md +34 -47
- package/src/skf-create-skill/references/component-extraction.md +8 -35
- package/src/skf-create-skill/references/ecosystem-check.md +9 -25
- package/src/skf-create-skill/references/enrich.md +4 -17
- package/src/skf-create-skill/references/extract.md +14 -29
- package/src/skf-create-skill/references/extraction-patterns-tracing.md +1 -1
- package/src/skf-create-skill/references/extraction-patterns.md +7 -7
- package/src/skf-create-skill/references/generate-artifacts.md +9 -23
- package/src/skf-create-skill/references/health-check.md +1 -1
- package/src/skf-create-skill/references/load-brief.md +16 -18
- package/src/skf-create-skill/references/report.md +12 -15
- package/src/skf-create-skill/references/source-resolution-protocols.md +3 -3
- package/src/skf-create-skill/references/step-auto-shard.md +52 -25
- package/src/skf-create-skill/references/step-doc-rot.md +30 -12
- package/src/skf-create-skill/references/step-doc-sources.md +2 -2
- package/src/skf-create-skill/references/sub/ccc-discover.md +2 -13
- package/src/skf-create-skill/references/sub/fetch-docs.md +8 -50
- package/src/skf-create-skill/references/sub/fetch-temporal.md +5 -20
- package/src/skf-create-skill/references/tier-degradation-rules.md +6 -8
- package/src/skf-create-skill/references/validate.md +59 -28
- package/src/skf-create-skill/scripts/scan-doc-rot.py +245 -0
- package/src/skf-create-stack-skill/SKILL.md +15 -9
- package/src/skf-create-stack-skill/assets/provenance-map-schema.md +1 -1
- package/src/skf-create-stack-skill/customize.toml +10 -0
- package/src/skf-create-stack-skill/references/compile-stack.md +7 -8
- package/src/skf-create-stack-skill/references/compose-mode-rules.md +2 -13
- package/src/skf-create-stack-skill/references/detect-integrations.md +49 -54
- package/src/skf-create-stack-skill/references/detect-manifests.md +37 -30
- package/src/skf-create-stack-skill/references/generate-output.md +24 -75
- package/src/skf-create-stack-skill/references/init.md +9 -81
- package/src/skf-create-stack-skill/references/manifest-patterns.md +5 -35
- package/src/skf-create-stack-skill/references/parallel-extract.md +12 -27
- package/src/skf-create-stack-skill/references/rank-and-confirm.md +6 -6
- package/src/skf-create-stack-skill/references/report.md +27 -62
- package/src/skf-create-stack-skill/references/validate.md +35 -31
- package/src/skf-drop-skill/SKILL.md +20 -23
- package/src/skf-drop-skill/customize.toml +10 -0
- package/src/skf-drop-skill/references/execute.md +51 -31
- package/src/skf-drop-skill/references/headless-contract.md +18 -0
- package/src/skf-drop-skill/references/health-check.md +1 -13
- package/src/skf-drop-skill/references/report.md +42 -10
- package/src/skf-drop-skill/references/select.md +88 -48
- package/src/skf-drop-skill/scripts/dir-sizes.py +116 -0
- package/src/skf-export-skill/SKILL.md +17 -14
- package/src/skf-export-skill/assets/managed-section-format.md +2 -5
- package/src/skf-export-skill/customize.toml +10 -6
- package/src/skf-export-skill/references/generate-snippet.md +21 -35
- package/src/skf-export-skill/references/load-skill.md +44 -42
- package/src/skf-export-skill/references/multi-skill-mode.md +1 -1
- package/src/skf-export-skill/references/orphan-context-detection.md +11 -3
- package/src/skf-export-skill/references/package.md +27 -33
- package/src/skf-export-skill/references/preflight-snippet-root-probe.md +24 -5
- package/src/skf-export-skill/references/result-envelope.md +28 -0
- package/src/skf-export-skill/references/summary.md +11 -5
- package/src/skf-export-skill/references/token-report.md +42 -24
- package/src/skf-export-skill/references/update-context.md +32 -33
- package/src/skf-forger/SKILL.md +27 -39
- package/src/skf-forger/references/pipeline-mode.md +34 -0
- package/src/skf-forger/scripts/parse-pipeline.py +270 -0
- package/src/skf-quick-skill/SKILL.md +10 -60
- package/src/skf-quick-skill/customize.toml +9 -0
- package/src/skf-quick-skill/references/batch-mode.md +6 -4
- package/src/skf-quick-skill/references/compile.md +15 -52
- package/src/skf-quick-skill/references/ecosystem-check.md +5 -20
- package/src/skf-quick-skill/references/finalize.md +12 -6
- package/src/skf-quick-skill/references/halt-contract.md +54 -0
- package/src/skf-quick-skill/references/health-check.md +3 -4
- package/src/skf-quick-skill/references/quick-extract.md +7 -18
- package/src/skf-quick-skill/references/registry-resolution.md +1 -23
- package/src/skf-quick-skill/references/resolve-target.md +27 -34
- package/src/skf-quick-skill/references/write-and-validate.md +10 -25
- package/src/skf-refine-architecture/SKILL.md +14 -7
- package/src/skf-refine-architecture/customize.toml +9 -0
- package/src/skf-refine-architecture/references/compile.md +9 -7
- package/src/skf-refine-architecture/references/gap-analysis.md +21 -50
- package/src/skf-refine-architecture/references/health-check.md +1 -1
- package/src/skf-refine-architecture/references/improvements.md +9 -25
- package/src/skf-refine-architecture/references/init.md +9 -16
- package/src/skf-refine-architecture/references/issue-detection.md +10 -25
- package/src/skf-refine-architecture/references/refinement-rules.md +4 -16
- package/src/skf-refine-architecture/references/report.md +13 -10
- package/src/skf-rename-skill/SKILL.md +11 -21
- package/src/skf-rename-skill/customize.toml +10 -0
- package/src/skf-rename-skill/references/execute.md +107 -78
- package/src/skf-rename-skill/references/exit-codes.md +12 -0
- package/src/skf-rename-skill/references/report.md +16 -8
- package/src/skf-rename-skill/references/select.md +69 -42
- package/src/skf-rename-skill/scripts/skf-validate-rename-name.py +161 -0
- package/src/skf-setup/SKILL.md +20 -12
- package/src/skf-setup/customize.toml +10 -0
- package/src/skf-setup/references/auto-index.md +9 -10
- package/src/skf-setup/references/ccc-index.md +14 -15
- package/src/skf-setup/references/detect-and-tier.md +6 -8
- package/src/skf-setup/references/report.md +22 -31
- package/src/skf-setup/references/tier-rules.md +2 -32
- package/src/skf-setup/references/write-config.md +9 -10
- package/src/skf-test-skill/SKILL.md +10 -8
- package/src/skf-test-skill/customize.toml +2 -0
- package/src/skf-test-skill/references/coherence-check.md +19 -31
- package/src/skf-test-skill/references/coverage-check.md +109 -75
- package/src/skf-test-skill/references/detect-mode.md +10 -41
- package/src/skf-test-skill/references/external-validators.md +11 -19
- package/src/skf-test-skill/references/init.md +38 -20
- package/src/skf-test-skill/references/migration-section-rules.md +4 -4
- package/src/skf-test-skill/references/report.md +60 -40
- package/src/skf-test-skill/references/score.md +32 -64
- package/src/skf-test-skill/references/scoring-rules.md +6 -62
- package/src/skf-test-skill/references/source-access-protocol.md +6 -6
- package/src/skf-test-skill/references/step-hard-gate.md +1 -3
- package/src/skf-test-skill/scripts/aggregate-coherence.py +276 -0
- package/src/skf-test-skill/scripts/check-metadata-coherence.py +359 -0
- package/src/skf-test-skill/scripts/combine-external-scores.py +190 -0
- package/src/skf-test-skill/scripts/compute-score.py +131 -1
- package/src/skf-test-skill/scripts/reconcile-coverage.py +327 -0
- package/src/skf-test-skill/scripts/validate-inventory.py +251 -0
- package/src/skf-test-skill/scripts/verify-declared-numerator.py +192 -0
- package/src/skf-update-skill/SKILL.md +17 -5
- package/src/skf-update-skill/customize.toml +14 -0
- package/src/skf-update-skill/references/detect-changes.md +19 -37
- package/src/skf-update-skill/references/health-check.md +3 -9
- package/src/skf-update-skill/references/init.md +39 -32
- package/src/skf-update-skill/references/manual-section-rules.md +1 -2
- package/src/skf-update-skill/references/merge-conflict-rules.md +1 -52
- package/src/skf-update-skill/references/merge.md +13 -57
- package/src/skf-update-skill/references/re-extract.md +12 -24
- package/src/skf-update-skill/references/report.md +14 -7
- package/src/skf-update-skill/references/validate.md +32 -30
- package/src/skf-update-skill/references/write.md +86 -51
- package/src/skf-update-skill/scripts/skf-new-file-diff.py +167 -0
- package/src/skf-verify-stack/SKILL.md +11 -21
- package/src/skf-verify-stack/customize.toml +12 -3
- package/src/skf-verify-stack/references/coverage-patterns.md +1 -29
- package/src/skf-verify-stack/references/coverage.md +21 -10
- package/src/skf-verify-stack/references/exit-codes.md +14 -0
- package/src/skf-verify-stack/references/health-check.md +1 -1
- package/src/skf-verify-stack/references/init.md +20 -25
- package/src/skf-verify-stack/references/integration-verification-rules.md +10 -19
- package/src/skf-verify-stack/references/integrations.md +44 -49
- package/src/skf-verify-stack/references/report.md +28 -34
- package/src/skf-verify-stack/references/requirements.md +3 -3
- package/src/skf-verify-stack/references/synthesize.md +36 -39
- package/src/skf-verify-stack/scripts/skf-coverage-tally.py +196 -0
- package/src/skf-verify-stack/scripts/skf-report-delta.py +256 -0
- package/src/skf-verify-stack/scripts/skf-verdict-rollup.py +258 -0
- package/src/skf-rename-skill/references/rebuild-context.md +0 -110
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
nextStepFile: 'detect-mode.md'
|
|
3
3
|
outputFile: '{forge_version}/test-report-{skill_name}-{run_id}.md'
|
|
4
|
-
templateFile: '
|
|
4
|
+
templateFile: '{testReportTemplatePath}'
|
|
5
5
|
sidecarFile: '{sidecar_path}/forge-tier.yaml'
|
|
6
6
|
skillsOutputFolder: '{skills_output_folder}'
|
|
7
7
|
# frontmatterScript resolves deterministically by probing two candidate
|
|
@@ -77,6 +77,12 @@ Check that the skill package contains required files:
|
|
|
77
77
|
|
|
78
78
|
This skill has not been created yet. Run the **create-skill** workflow first."
|
|
79
79
|
|
|
80
|
+
**Headless envelope (if `{headless_mode}`):** emit to **stderr**:
|
|
81
|
+
|
|
82
|
+
```
|
|
83
|
+
SKF_TEST_RESULT_JSON: {"status":"error","skill_name":"{skill_name}","verdict":null,"score":null,"threshold":null,"report_path":null,"next_workflow":null,"exit_code":1,"halt_reason":"target-inaccessible"}
|
|
84
|
+
```
|
|
85
|
+
|
|
80
86
|
HALT — do not proceed.
|
|
81
87
|
|
|
82
88
|
**If metadata.json missing:**
|
|
@@ -89,9 +95,9 @@ HALT — do not proceed.
|
|
|
89
95
|
1. `{project-root}/_bmad/skf/shared/scripts/skf-validate-frontmatter.py` (installed module layout)
|
|
90
96
|
2. `{project-root}/src/shared/scripts/skf-validate-frontmatter.py` (development-tree layout)
|
|
91
97
|
|
|
92
|
-
Use the
|
|
98
|
+
Use the first path that exists as `{frontmatterScript}`. There is no manual fallback.
|
|
93
99
|
|
|
94
|
-
**If neither path exists, HALT** with the diagnostic below. test-skill is a quality gate; without the deterministic validator it cannot produce a trustworthy frontmatter verdict, and silent manual
|
|
100
|
+
**If neither path exists, HALT** with the diagnostic below. test-skill is a quality gate; without the deterministic validator it cannot produce a trustworthy frontmatter verdict, and a silent manual check can miss subtle spec drift. The missing helper must be restored before testing continues:
|
|
95
101
|
|
|
96
102
|
```
|
|
97
103
|
Error: cannot locate skf-validate-frontmatter.py at either of:
|
|
@@ -104,7 +110,9 @@ SKF module (`skf init`) or run from a development checkout with src/ present.
|
|
|
104
110
|
|
|
105
111
|
Do not proceed. No partial test report is written.
|
|
106
112
|
|
|
107
|
-
**3b.
|
|
113
|
+
**3b. Python runtime probe (before invoking the validator).** Confirm both `python3` and `uv` are on `$PATH` (`command -v python3` and `command -v uv`). Both are required: `uv run` shells through to `python3` and honors the script's PEP 723 PyYAML dependency declaration that bare `python3` ignores (bare `python3` fails with `ModuleNotFoundError: No module named 'yaml'` on a fresh interpreter — `docs/getting-started.md` documents uv as the runtime prereq for exactly this). If either is missing, set `analysis_confidence: degraded` in workflow context and carry a **score cap** into step 5: `capped_score = threshold - 1` → forces auto-FAIL until the runtime is restored. Record the reason in evidence-report and the test report frontmatter (`analysisConfidence: degraded`, `toolingStatus: python3-missing` or `uv-missing` as appropriate). `uv` is a documented runtime prerequisite — see `docs/getting-started.md` for install instructions.
|
|
114
|
+
|
|
115
|
+
**3c. Run the validator (30s timeout — the deterministic validator should finish in <1s; the cap only guards against runaway python).**
|
|
108
116
|
|
|
109
117
|
```bash
|
|
110
118
|
timeout 30s uv run {frontmatterScript} {resolved_skill_package}/SKILL.md --skill-dir-name {skill_name}
|
|
@@ -119,7 +127,11 @@ Parse the JSON output. Treat each `status` value explicitly:
|
|
|
119
127
|
|
|
120
128
|
- `status: "pass"` — continue silently.
|
|
121
129
|
- `status: "warn"` — display the warning below, log each issue as a pre-check finding, and continue with testing. Frontmatter issues surface in the gap report alongside coverage/coherence findings.
|
|
122
|
-
- `status: "fail"` — **HALT with auto-FAIL.** Frontmatter failure
|
|
130
|
+
- `status: "fail"` — **HALT with auto-FAIL.** Frontmatter failure means the skill will be rejected by `npx skills add` and `npx skill-check check`; shipping it would produce a false PASS downstream. Write the halt note into evidence-report and exit non-zero. **Headless envelope (if `{headless_mode}`):** emit to **stderr** before halting. The output document does not exist yet (created in §6), so `report_path` is `null` — matching the other pre-report init HALTs (target-inaccessible, forge-tier-missing, workspace-drift, another-run-active). A frontmatter-invalid target is the most common failure this gate exists to catch, so a headless orchestrator must be able to branch on it (route to update-skill) rather than see an unlabelled non-zero exit:
|
|
131
|
+
|
|
132
|
+
```
|
|
133
|
+
SKF_TEST_RESULT_JSON: {"status":"error","skill_name":"{skill_name}","verdict":null,"score":null,"threshold":null,"report_path":null,"next_workflow":null,"exit_code":1,"halt_reason":"frontmatter-invalid"}
|
|
134
|
+
```
|
|
123
135
|
|
|
124
136
|
```
|
|
125
137
|
**Warning/Error: SKILL.md frontmatter is non-compliant with agentskills.io specification.**
|
|
@@ -129,8 +141,6 @@ Parse the JSON output. Treat each `status` value explicitly:
|
|
|
129
141
|
This skill will fail `npx skills add` and `npx skill-check check`. {If warn:} Consider fixing frontmatter before proceeding (run `npx skill-check check <skill-dir> --fix` to auto-fix deterministic issues). {If fail:} test-skill cannot proceed — halt and repair frontmatter, then re-run.
|
|
130
142
|
```
|
|
131
143
|
|
|
132
|
-
**3c. Python runtime probe.** Before the first invocation, confirm both `python3` AND `uv` are on `$PATH` (`command -v python3` and `command -v uv`). Both are required: `uv run` shells through to `python3` under the hood AND honors the script's PEP 723 PyYAML dependency declaration that bare `python3` ignores. If either is missing, set `analysis_confidence: degraded` in workflow context and carry a **score cap** into step 5: `capped_score = threshold - 1` → forces auto-FAIL until the runtime is restored. Record the reason in evidence-report and the test report frontmatter (`analysisConfidence: degraded`, `toolingStatus: python3-missing` or `uv-missing` as appropriate). `uv` is a documented runtime prerequisite — see `docs/getting-started.md` for install instructions.
|
|
133
|
-
|
|
134
144
|
### 4. Load Forge Tier State
|
|
135
145
|
|
|
136
146
|
**`--tier=<...>` flag bypass (precedes the sidecar probe).** If `tier_flag` is set in workflow context (from §1's `--tier=<Quick|Forge|Forge+|Deep>` flag), validate the value against the allowed set. On valid match: set `detected_tier` directly to the flag's value, leave `ast_grep`/`gh_cli`/`qmd` availability flags unset (downstream steps treat unset as "unknown" — analysis proceeds without tool-specific enrichment), log Info note "tier — supplied via --tier flag, sidecar bypassed", and SKIP the sidecar probe and HALT below (jump straight to §4b "Apply Tier Override"). On invalid value (not one of the four), HALT with "Error: --tier=<value> is not one of Quick, Forge, Forge+, Deep".
|
|
@@ -144,6 +154,12 @@ This skill will fail `npx skills add` and `npx skill-check check`. {If warn:} Co
|
|
|
144
154
|
**If forge-tier.yaml missing:**
|
|
145
155
|
"**Cannot proceed.** forge-tier.yaml not found at `{sidecarFile}`. Please run the **setup** workflow first to configure your forge tier (Quick/Forge/Forge+/Deep), or re-run with `--tier=<Quick|Forge|Forge+|Deep>` to bypass the sidecar."
|
|
146
156
|
|
|
157
|
+
**Headless envelope (if `{headless_mode}`):** emit to **stderr**:
|
|
158
|
+
|
|
159
|
+
```
|
|
160
|
+
SKF_TEST_RESULT_JSON: {"status":"error","skill_name":"{skill_name}","verdict":null,"score":null,"threshold":null,"report_path":null,"next_workflow":null,"exit_code":1,"halt_reason":"forge-tier-missing"}
|
|
161
|
+
```
|
|
162
|
+
|
|
147
163
|
HALT — do not proceed.
|
|
148
164
|
|
|
149
165
|
### 4b. Apply Tier Override (if set)
|
|
@@ -169,11 +185,11 @@ Test-skill reads `source_path` during coverage and coherence analysis. If the lo
|
|
|
169
185
|
|
|
170
186
|
- Resolve `pinned_commit` from `metadata.source_commit`.
|
|
171
187
|
- **If `pinned_commit` is null, empty, or `"local"`:** skip the guard; log `workspace_drift_check: skipped (no pinned commit)` and continue to section 6.
|
|
172
|
-
- **If `pinned_commit` is a per-repo map (stack skills):** iterate each `{repo_path: commit}` entry — for each repo run `git -C "{repo_path}" rev-parse HEAD` and compare to its pinned commit (accept full-SHA or short-SHA-prefix match). If ANY repo diverges and the user did not pass `--allow-workspace-drift`, HALT with exit status `
|
|
188
|
+
- **If `pinned_commit` is a per-repo map (stack skills):** iterate each `{repo_path: commit}` entry — for each repo run `git -C "{repo_path}" rev-parse HEAD` and compare to its pinned commit (accept full-SHA or short-SHA-prefix match). If ANY repo diverges and the user did not pass `--allow-workspace-drift`, HALT with exit status `workspace-drift` listing every mismatched repo (in `{headless_mode}`, emit the same `workspace-drift` stderr envelope shown in the single-tree branch below before halting). On all-match: log `workspace_drift_check: ok (stack, {N} repos verified)` and continue to section 6. This guard must iterate every repo — do not skip stack skills.
|
|
173
189
|
- **If `source_path` is not a git working tree** (bare checkout, tarball extract, docs-only source) — detect by `git -C "{source_path}" rev-parse --is-inside-work-tree`, non-zero exit means skip: log `workspace_drift_check: skipped (not a git working tree)` and continue to section 6.
|
|
174
190
|
- **Otherwise** run `git -C "{source_path}" rev-parse HEAD` and compare to `pinned_commit`. Accept full-SHA or short-SHA-prefix match (stored pins are often 8-char short hashes — see `src/knowledge/provenance-tracking.md`).
|
|
175
191
|
- **On match:** log `workspace_drift_check: ok ({short_sha})` and continue.
|
|
176
|
-
- **On mismatch, AND the user did not pass `--allow-workspace-drift`:** HALT with exit status `
|
|
192
|
+
- **On mismatch, AND the user did not pass `--allow-workspace-drift`:** HALT with exit status `workspace-drift`. Display:
|
|
177
193
|
|
|
178
194
|
```
|
|
179
195
|
Workspace HEAD does not match the commit this skill was pinned against.
|
|
@@ -191,14 +207,24 @@ Test-skill reads `source_path` during coverage and coherence analysis. If the lo
|
|
|
191
207
|
current workspace (accepts that findings reflect HEAD, not the pin).
|
|
192
208
|
```
|
|
193
209
|
|
|
210
|
+
**Headless envelope (if `{headless_mode}`):** emit to **stderr**:
|
|
211
|
+
|
|
212
|
+
```
|
|
213
|
+
SKF_TEST_RESULT_JSON: {"status":"error","skill_name":"{skill_name}","verdict":null,"score":null,"threshold":null,"report_path":null,"next_workflow":null,"exit_code":1,"halt_reason":"workspace-drift"}
|
|
214
|
+
```
|
|
215
|
+
|
|
194
216
|
Do not proceed. The test report has not been created; no partial writes.
|
|
195
217
|
- **On mismatch WITH `--allow-workspace-drift`:** log `workspace_drift_check: overridden (pinned={pinned_commit}, head={head_sha})`, carry the warning into the final report frontmatter (`workspaceDrift: overridden`), and set `allow_workspace_drift: true` in workflow context (consumed by step 5 §5 drift override — a PASS under drift is demoted to `pass-with-drift` and `nextWorkflow` is forced to `update-skill`, never `export-skill`). Continue.
|
|
196
218
|
|
|
197
219
|
### 6. Create Output Document
|
|
198
220
|
|
|
199
|
-
**6a. Generate `{run_id}`**: a per-run identifier of the form `{YYYYMMDDTHHmmssZ}-{pid}-{rand4}` (UTC timestamp + process PID + 4-char random hex). Store in workflow context. All per-run artifacts in this and subsequent steps
|
|
221
|
+
**6a. Generate `{run_id}`**: a per-run identifier of the form `{YYYYMMDDTHHmmssZ}-{pid}-{rand4}` (UTC timestamp + process PID + 4-char random hex). Store in workflow context. All per-run artifacts in this and subsequent steps must carry this suffix; step 6 verifies `testDate` in the resulting report matches the run's stamp and fail-fast otherwise.
|
|
200
222
|
|
|
201
|
-
**6b. Acquire the per-skill test lock**: `flock {forge_version}/.test-skill.lock` for the duration of this run to serialize concurrent `skf-test-skill` invocations against the same skill. If the lock is already held by another run, HALT with "another test-skill run is active for {skill_name}".
|
|
223
|
+
**6b. Acquire the per-skill test lock**: `flock {forge_version}/.test-skill.lock` for the duration of this run to serialize concurrent `skf-test-skill` invocations against the same skill. If the lock is already held by another run, HALT with "another test-skill run is active for {skill_name}". **Headless envelope (if `{headless_mode}`):** emit to **stderr** before halting:
|
|
224
|
+
|
|
225
|
+
```
|
|
226
|
+
SKF_TEST_RESULT_JSON: {"status":"error","skill_name":"{skill_name}","verdict":null,"score":null,"threshold":null,"report_path":null,"next_workflow":null,"exit_code":1,"halt_reason":"another-run-active"}
|
|
227
|
+
```
|
|
202
228
|
|
|
203
229
|
**6c. Create `{outputFile}` from `{templateFile}`** — use `{forge_version}/test-report-{skill_name}-{run_id}.md` Initial frontmatter:
|
|
204
230
|
|
|
@@ -224,15 +250,7 @@ nextWorkflow: ''
|
|
|
224
250
|
|
|
225
251
|
### 7. Report Initialization Status
|
|
226
252
|
|
|
227
|
-
|
|
228
|
-
|
|
229
|
-
**Skill:** {skill_name}
|
|
230
|
-
**Path:** {skill_path}
|
|
231
|
-
**Type:** {skill_type}
|
|
232
|
-
**Forge Tier:** {detected_tier}
|
|
233
|
-
**Source:** {source_path}
|
|
234
|
-
|
|
235
|
-
**Proceeding to mode detection...**"
|
|
253
|
+
Report initialization to the user: the resolved skill name, path, type, forge tier, and source path. Then proceed to mode detection.
|
|
236
254
|
|
|
237
255
|
Update stepsCompleted, then load and execute {nextStepFile}.
|
|
238
256
|
|
|
@@ -8,7 +8,7 @@
|
|
|
8
8
|
|
|
9
9
|
## Gate Check
|
|
10
10
|
|
|
11
|
-
Execute this check
|
|
11
|
+
Execute this check only when both conditions are met:
|
|
12
12
|
|
|
13
13
|
1. Forge tier is **Deep** (tool-gated)
|
|
14
14
|
2. `{forge_data_folder}/{skill_name}/evidence-report.md` exists (data-gated)
|
|
@@ -48,7 +48,7 @@ Check whether SKILL.md contains a "Migration & Deprecation Warnings" section
|
|
|
48
48
|
pinned `t2_future_count` field — this is the authoritative count, not the
|
|
49
49
|
narrative body.
|
|
50
50
|
|
|
51
|
-
**Detection contract
|
|
51
|
+
**Detection contract.** Read the frontmatter deterministically:
|
|
52
52
|
|
|
53
53
|
```bash
|
|
54
54
|
# Extract t2_future_count from frontmatter. Requires a `---` delimiter pair.
|
|
@@ -57,7 +57,7 @@ awk '/^---$/{c++;next} c==1 && /^t2_future_count:/{print $2; exit}' \
|
|
|
57
57
|
```
|
|
58
58
|
|
|
59
59
|
- **Frontmatter missing OR `t2_future_count` absent** → treat as Case 4 (see
|
|
60
|
-
below) and skip silently. Do
|
|
60
|
+
below) and skip silently. Do not fall back to grepping prose (`grep "T2-future"`) —
|
|
61
61
|
prose drift (heading renames, alt phrasings like "forward-looking
|
|
62
62
|
annotations", capitalization variance) silently breaks detection and can
|
|
63
63
|
invert Case-1 vs Case-2/3 severity.
|
|
@@ -65,7 +65,7 @@ awk '/^---$/{c++;next} c==1 && /^t2_future_count:/{print $2; exit}' \
|
|
|
65
65
|
below.
|
|
66
66
|
|
|
67
67
|
The pinned field is emitted by `skf-create-skill/references/compile.md`
|
|
68
|
-
§7 (frontmatter-pinned fields), which
|
|
68
|
+
§7 (frontmatter-pinned fields), which always writes `t2_future_count: N`
|
|
69
69
|
(including 0). Legacy skills whose `evidence-report.md` predates the pinned
|
|
70
70
|
field land in Case 4.
|
|
71
71
|
|
|
@@ -2,8 +2,8 @@
|
|
|
2
2
|
nextStepFile: 'health-check.md'
|
|
3
3
|
|
|
4
4
|
outputFile: '{forge_version}/test-report-{skill_name}-{run_id}.md'
|
|
5
|
-
scoringRulesFile: '
|
|
6
|
-
outputFormatsFile: '
|
|
5
|
+
scoringRulesFile: '{scoringRulesPath}'
|
|
6
|
+
outputFormatsFile: '{outputFormatsPath}'
|
|
7
7
|
# outputContractSchema and healthCheck resolve relative to the SKF module root
|
|
8
8
|
# (`{project-root}/_bmad/skf/` when installed, `{project-root}/src/` during
|
|
9
9
|
# development), NOT relative to this step file. Both paths are probed in
|
|
@@ -23,7 +23,7 @@ atomicWriteProbeOrder:
|
|
|
23
23
|
|
|
24
24
|
## STEP GOAL:
|
|
25
25
|
|
|
26
|
-
Generate a detailed gap report listing every issue found during coverage and coherence analysis, assign severity to each gap, provide specific actionable remediation suggestions, and finalize the test report document. Do not recalculate scores — that ran in step 5. This step chains to the local health-check step via `{nextStepFile}` after completion; the user-facing report is
|
|
26
|
+
Generate a detailed gap report listing every issue found during coverage and coherence analysis, assign severity to each gap, provide specific actionable remediation suggestions, and finalize the test report document. Do not recalculate scores — that ran in step 5. This step chains to the local health-check step via `{nextStepFile}` after completion; the user-facing report is not the terminal step.
|
|
27
27
|
|
|
28
28
|
### 1. Collect All Issues
|
|
29
29
|
|
|
@@ -46,27 +46,13 @@ Read `{outputFile}` and extract every issue found across all analysis sections:
|
|
|
46
46
|
|
|
47
47
|
### 2. Load Severity Rules
|
|
48
48
|
|
|
49
|
-
Load `{scoringRulesFile}` for gap
|
|
50
|
-
|
|
51
|
-
| Severity | Criteria |
|
|
52
|
-
|----------|----------|
|
|
53
|
-
| **Critical** | Missing exported function/class documentation |
|
|
54
|
-
| **High** | Signature mismatch between source and SKILL.md |
|
|
55
|
-
| **Medium** | Missing type or interface documentation |
|
|
56
|
-
| **Medium** | Migration section present/absent mismatch with T2-future annotation data (Deep tier) |
|
|
57
|
-
| **Medium** | Script/asset directory exists but no Scripts & Assets section in SKILL.md |
|
|
58
|
-
| **Medium** | Scripts & Assets section references file not found in scripts/ or assets/ directory |
|
|
59
|
-
| **Low** | Script/asset file present without provenance entry in provenance-map.json file_entries |
|
|
60
|
-
| **Low** | Missing optional metadata or examples |
|
|
61
|
-
| **Low** | Description trigger optimization recommended (third-person voice, negative triggers, or keyword coverage gaps) |
|
|
62
|
-
| **Info** | Style suggestions, non-blocking observations |
|
|
63
|
-
| **Info** | Discovery testing not performed — realistic prompt testing recommended before export |
|
|
49
|
+
Load the **Gap Severity** table from `{scoringRulesFile}` — it is the single source of truth for classifying every gap (Critical / High / Medium / Low / Info). Classify each issue directly against that table in §3; do not restate its rows here, so the table cannot drift between this step and the scoring step that already consumes it.
|
|
64
50
|
|
|
65
51
|
### 3. Classify and Order Gaps
|
|
66
52
|
|
|
67
53
|
Load `{outputFormatsFile}` for gap entry format and remediation quality rules.
|
|
68
54
|
|
|
69
|
-
For each issue, assign severity from `{scoringRulesFile}` and generate a specific remediation following the quality rules in `{outputFormatsFile}`. Remediation suggestions
|
|
55
|
+
For each issue, assign severity from `{scoringRulesFile}` and generate a specific remediation following the quality rules in `{outputFormatsFile}`. Remediation suggestions must reference specific files, exports, and line numbers. Order gaps by severity: Critical → High → Medium → Low → Info.
|
|
70
56
|
|
|
71
57
|
### 4. Generate Remediation Summary and Append Gap Report
|
|
72
58
|
|
|
@@ -74,11 +60,11 @@ Load the Gap Report section format from `{outputFormatsFile}`. Count gaps by sev
|
|
|
74
60
|
|
|
75
61
|
If no gaps found, append a clean pass message recommending **export-skill** workflow.
|
|
76
62
|
|
|
77
|
-
### 4b. Discovery Testing
|
|
63
|
+
### 4b. Discovery Testing
|
|
78
64
|
|
|
79
65
|
**`--no-discovery` flag bypass (precedes the precondition check).** If `no_discovery: true` is set in workflow context (from §1 of `init.md` — `--no-discovery` flag on invocation), record an Info-severity note in the Discovery Quality subsection: `discovery — skipped: --no-discovery flag set`, log the bypass, and SKIP §4b.1–§4b.3. Proceed to §4b.4 (description optimization) only if tessl/skill-check flagged description issues; otherwise skip directly to §4c.
|
|
80
66
|
|
|
81
|
-
After gap enumeration, perform minimum-viable discovery testing. This is a **Medium-weight** check contributing to the Discovery Quality subsection
|
|
67
|
+
After gap enumeration, perform minimum-viable discovery testing. This is a **Medium-weight** check contributing to the Discovery Quality subsection.
|
|
82
68
|
|
|
83
69
|
**4b.0 Precondition — catalog size check:**
|
|
84
70
|
|
|
@@ -100,13 +86,15 @@ If SKILL.md does not contain enough organic examples, synthesize 3 from the skil
|
|
|
100
86
|
|
|
101
87
|
**4b.2 Spawn a discovery subagent:**
|
|
102
88
|
|
|
89
|
+
**Subagents-unavailable guard (precedes the spawn).** If subagents cannot be spawned in this environment (e.g. a headless/CI pipeline with no subagent capability), do not fall back to answering the routing in the main thread — the main thread knows which skill is under test, so it would self-route to `3/3 PASS` and inflate the Discovery score, the exact false confidence §4b.0 warns against. Instead record an Info-severity note in the Discovery Quality subsection: `discovery — skipped: subagents unavailable, routing test requires isolated context`, exclude the discovery check from Discovery Quality scoring (do not count it PASS or FAIL), and skip to §4c.
|
|
90
|
+
|
|
103
91
|
For each of the 3 prompts, spawn an isolated subagent with NO prior context about which skill is under test. Provide only:
|
|
104
92
|
1. A compact list of ALL skills available in `{skillsOutputFolder}` (name + description line from each skill's SKILL.md frontmatter)
|
|
105
93
|
2. The prompt text
|
|
106
94
|
|
|
107
95
|
Instruction to the subagent:
|
|
108
96
|
|
|
109
|
-
> "You are an agent selecting the best skill to handle a user request. Here is the catalog: {catalog}. The user says: '{prompt}'. Return JSON: `{\"selected_skill\": \"<name>\", \"confidence\": \"<high|medium|low>\", \"reasoning\": \"<one sentence>\"}`. If no skill fits, return `{\"selected_skill\": null, ...}`. Return
|
|
97
|
+
> "You are an agent selecting the best skill to handle a user request. Here is the catalog: {catalog}. The user says: '{prompt}'. Return JSON: `{\"selected_skill\": \"<name>\", \"confidence\": \"<high|medium|low>\", \"reasoning\": \"<one sentence>\"}`. If no skill fits, return `{\"selected_skill\": null, ...}`. Return only JSON."
|
|
110
98
|
|
|
111
99
|
**4b.3 Evaluate discovery results:**
|
|
112
100
|
|
|
@@ -133,13 +121,17 @@ Realistic prompt patterns for synthesis (§4b.1 fallback):
|
|
|
133
121
|
|
|
134
122
|
### 4c. Result Contract (atomic write)
|
|
135
123
|
|
|
136
|
-
**Resolve `{atomicWriteHelper}`:** probe `{atomicWriteProbeOrder}`. HALT if neither candidate exists — the contract is a downstream-consumer protocol and must never be written non-atomically.
|
|
124
|
+
**Resolve `{atomicWriteHelper}`:** probe `{atomicWriteProbeOrder}`. HALT if neither candidate exists — the contract is a downstream-consumer protocol and must never be written non-atomically. **Headless envelope (if `{headless_mode}`):** emit to **stderr** before halting:
|
|
125
|
+
|
|
126
|
+
```
|
|
127
|
+
SKF_TEST_RESULT_JSON: {"status":"error","skill_name":"{skill_name}","verdict":null,"score":null,"threshold":null,"report_path":"{outputFile}","next_workflow":null,"exit_code":1,"halt_reason":"atomic-writer-missing"}
|
|
128
|
+
```
|
|
137
129
|
|
|
138
130
|
Write the result contract per `{outputContractSchema}`:
|
|
139
131
|
- Per-run record: `{forge_version}/skf-test-skill-result-{run_id}.json` (the `{run_id}` set in step 1 §6a — already carries UTC timestamp + PID + random suffix, so no same-second collision).
|
|
140
132
|
- Latest copy: `{forge_version}/skf-test-skill-result-latest.json` (stable path for pipeline consumers — copy, not symlink).
|
|
141
133
|
|
|
142
|
-
Both writes
|
|
134
|
+
Both writes must go through the atomic writer so partial writes are never observable:
|
|
143
135
|
|
|
144
136
|
```bash
|
|
145
137
|
# Build the JSON payload in memory, then:
|
|
@@ -149,7 +141,7 @@ cat payload.json | python3 {atomicWriteHelper} write --target {forge_version}/sk
|
|
|
149
141
|
|
|
150
142
|
Payload contents:
|
|
151
143
|
- `outputs[]` — include the test report path at `{outputFile}` with its `{run_id}` suffix
|
|
152
|
-
- `summary` — `score`, `threshold`, `result` (`"PASS"`, `"PASS_WITH_DRIFT"`, `"FAIL"`, or **`"INCONCLUSIVE"`**), `testMode` (naive/contextual), `activeCategories[]`, `inconclusiveReasons[]` (when present). `PASS_WITH_DRIFT` is set when the workflow observed workspace drift and the user passed `--allow-workspace-drift` — see step 5 §5 drift override. Downstream consumers
|
|
144
|
+
- `summary` — `score`, `threshold`, `result` (`"PASS"`, `"PASS_WITH_DRIFT"`, `"FAIL"`, or **`"INCONCLUSIVE"`**), `testMode` (naive/contextual), `activeCategories[]`, `inconclusiveReasons[]` (when present). `PASS_WITH_DRIFT` is set when the workflow observed workspace drift and the user passed `--allow-workspace-drift` — see step 5 §5 drift override. Downstream consumers must treat `PASS_WITH_DRIFT` as a non-exportable result: re-run against the pinned commit before export. When threshold fallback occurred, add `threshold_fallback: true`, `original_threshold: {N}`, and `evidence_report_path: '{path}'` to the summary — these fields are absent (not `false`/`null`) when no fallback occurred.
|
|
153
145
|
- `runId` — the workflow's `{run_id}` for downstream correlation
|
|
154
146
|
- `healthCheckDispatched` — boolean, set by §7 after the dispatch decision
|
|
155
147
|
|
|
@@ -178,13 +170,19 @@ Run it with a bounded timeout (default 60s). On success: log Info note "on_compl
|
|
|
178
170
|
'report']
|
|
179
171
|
```
|
|
180
172
|
|
|
181
|
-
If any expected entry is missing, HALT with "step completeness violation — missing {list}; workflow state is inconsistent, do not finalize the report".
|
|
173
|
+
If any expected entry is missing, HALT with "step completeness violation — missing {list}; workflow state is inconsistent, do not finalize the report". **Headless envelope (if `{headless_mode}`):** emit to **stderr** before halting:
|
|
174
|
+
|
|
175
|
+
```
|
|
176
|
+
SKF_TEST_RESULT_JSON: {"status":"error","skill_name":"{skill_name}","verdict":null,"score":null,"threshold":null,"report_path":"{outputFile}","next_workflow":null,"exit_code":1,"halt_reason":"step-completeness-violation"}
|
|
177
|
+
```
|
|
178
|
+
|
|
179
|
+
Only append `'report'` and write back after the check passes.
|
|
182
180
|
|
|
183
181
|
**Section anchor presence check (companion to stepsCompleted).** The
|
|
184
182
|
report template ships six canonical H2 anchors — one per populating step. An
|
|
185
183
|
off-sequence run (e.g. a step wrote its section into the wrong anchor, or a
|
|
186
184
|
subagent truncated the file) can leave `stepsCompleted` intact while a section
|
|
187
|
-
is missing. `grep -n` each anchor below against `{outputFile}`; each
|
|
185
|
+
is missing. `grep -n` each anchor below against `{outputFile}`; each must
|
|
188
186
|
return ≥1 match:
|
|
189
187
|
|
|
190
188
|
```
|
|
@@ -197,12 +195,18 @@ return ≥1 match:
|
|
|
197
195
|
```
|
|
198
196
|
|
|
199
197
|
On any miss, HALT with "report anchor missing: {anchor} — section was not
|
|
200
|
-
appended by its owning step".
|
|
201
|
-
|
|
198
|
+
appended by its owning step". **Headless envelope (if `{headless_mode}`):** emit to **stderr** before halting:
|
|
199
|
+
|
|
200
|
+
```
|
|
201
|
+
SKF_TEST_RESULT_JSON: {"status":"error","skill_name":"{skill_name}","verdict":null,"score":null,"threshold":null,"report_path":"{outputFile}","next_workflow":null,"exit_code":1,"halt_reason":"report-anchor-missing"}
|
|
202
|
+
```
|
|
203
|
+
|
|
204
|
+
Do not append `'report'` and do not
|
|
205
|
+
write the result contract. The template at `{testReportTemplatePath}`
|
|
202
206
|
declares these anchors as TBD placeholders; a miss means a step silently
|
|
203
207
|
skipped its append.
|
|
204
208
|
|
|
205
|
-
**INCONCLUSIVE as gate:** if `testResult == 'inconclusive'` (from step 5), the report final presentation (§6) and result contract (§4c) have already been written with that verdict. Do
|
|
209
|
+
**INCONCLUSIVE as gate:** if `testResult == 'inconclusive'` (from step 5), the report final presentation (§6) and result contract (§4c) have already been written with that verdict. Do not auto-map INCONCLUSIVE to PASS or FAIL. Recommend `manual-review`. The step must still complete (health-check runs unconditionally) — INCONCLUSIVE is a report-time signal, not a workflow abort.
|
|
206
210
|
|
|
207
211
|
### 6. Present Final Report
|
|
208
212
|
|
|
@@ -238,7 +242,7 @@ skipped its append.
|
|
|
238
242
|
**update-skill** — This skill needs remediation. Review the gap report above and run the update-skill workflow to address the {N} blocking issues (Critical + High).
|
|
239
243
|
|
|
240
244
|
{IF INCONCLUSIVE:}
|
|
241
|
-
**manual-review** — The evidence base was too thin to grade automatically. See `inconclusiveReasons` in the Completeness Score section. Typical fixes: upgrade forge tier, enable external validators, or re-extract with a wider scope. Do
|
|
245
|
+
**manual-review** — The evidence base was too thin to grade automatically. See `inconclusiveReasons` in the Completeness Score section. Typical fixes: upgrade forge tier, enable external validators, or re-extract with a wider scope. Do not export.
|
|
242
246
|
|
|
243
247
|
---
|
|
244
248
|
|
|
@@ -246,19 +250,29 @@ skipped its append.
|
|
|
246
250
|
|
|
247
251
|
**Test report finalized.**"
|
|
248
252
|
|
|
249
|
-
### 6b. Headless Exit
|
|
253
|
+
### 6b. Determine Headless Exit Code
|
|
254
|
+
|
|
255
|
+
This step only determines the terminal exit code — it does not exit. Both modes then reach §7 (headless auto-proceeds past the menu; non-headless goes through the [C] menu), and the terminal process-exit with this code happens in §7 after the health-check dispatch.
|
|
256
|
+
|
|
257
|
+
If `{headless_mode}`, map `testResult` to the code the workflow will exit with in §7 and store it as `{headless_exit_code}` in workflow context:
|
|
258
|
+
- `testResult: 'pass'` → exit code 0
|
|
259
|
+
- `testResult: 'pass-with-drift'` → exit code 4 (distinct from clean pass — see the pass-with-drift row in SKILL.md Exit Codes; exiting 0 under a drift override would wrongly signal a clean pass)
|
|
260
|
+
- `testResult: 'fail'` → exit code 2 (the result contract was written in §4c — never exit before it)
|
|
261
|
+
- `testResult: 'inconclusive'` → exit code 3 (distinct from fail so orchestrators can route to manual-review queues)
|
|
250
262
|
|
|
251
|
-
|
|
252
|
-
- `testResult: 'pass'` → `exit 0`
|
|
253
|
-
- `testResult: 'pass-with-drift'` → `exit 4` (distinct from clean pass — orchestrators MUST route to re-test-against-pinned-commit queues and refuse export; never exit 0 under drift override)
|
|
254
|
-
- `testResult: 'fail'` → `exit 2` (after the result contract has been written in §4c — never before)
|
|
255
|
-
- `testResult: 'inconclusive'` → `exit 3` (distinct from fail so orchestrators can route to manual-review queues)
|
|
263
|
+
### 6c. Emit Headless Result Envelope (stdout)
|
|
256
264
|
|
|
257
|
-
|
|
265
|
+
If `{headless_mode}`, emit the terminal result envelope to **stdout** as a single line before chaining to §7 — this is the branchable record a headless orchestrator reads for the happy path (PASS / FAIL / INCONCLUSIVE / pass-with-drift). The SKILL.md Result Contract owns the shape and the field rules; the on-disk copy written in §4c is the richer form. Build it from the settled verdict and the values already in the output frontmatter:
|
|
266
|
+
|
|
267
|
+
```
|
|
268
|
+
SKF_TEST_RESULT_JSON: {"status":"success","skill_name":"{skill_name}","verdict":"{PASS|FAIL|INCONCLUSIVE|pass-with-drift}","score":{score},"threshold":{threshold},"report_path":"{outputFile}","next_workflow":{export-skill when PASS | update-skill when FAIL or pass-with-drift | null when INCONCLUSIVE},"exit_code":{headless_exit_code},"halt_reason":null}
|
|
269
|
+
```
|
|
270
|
+
|
|
271
|
+
`verdict` is uppercase for `pass`/`fail`/`inconclusive` (→ `PASS`/`FAIL`/`INCONCLUSIVE`) and the literal `pass-with-drift`. When threshold fallback occurred (frontmatter `thresholdFallback: true`), add `"threshold_fallback":true` and `"original_threshold":{originalThreshold}`; omit both otherwise. Non-headless runs skip this emission — the §6 presentation is their terminal output.
|
|
258
272
|
|
|
259
273
|
### 7. Health-Check Dispatch + MENU OPTIONS
|
|
260
274
|
|
|
261
|
-
**`--no-health-check` flag bypass (precedes the health-check resolution).** If `no_health_check: true` is set in workflow context (from §1 of `init.md` — `--no-health-check` flag on invocation), set `health_check_dispatched: false` in the output report frontmatter and mirror `healthCheckDispatched: false` into the result contract written in §4c (re-write atomically via `{atomicWriteHelper}`). Log Info note "health-check — skipped: --no-health-check flag set" and
|
|
275
|
+
**`--no-health-check` flag bypass (precedes the health-check resolution).** If `no_health_check: true` is set in workflow context (from §1 of `init.md` — `--no-health-check` flag on invocation), set `health_check_dispatched: false` in the output report frontmatter and mirror `healthCheckDispatched: false` into the result contract written in §4c (re-write atomically via `{atomicWriteHelper}`). Log Info note "health-check — skipped: --no-health-check flag set" and exit the workflow: in `{headless_mode}`, exit with `{headless_exit_code}` (determined in §6b); non-headless, simply terminate after the §6 presentation. Do not resolve `{healthCheckFile}`, do not display the menu, do not chain to `{nextStepFile}`. This flag is the one path where §7 does not dispatch the health-check.
|
|
262
276
|
|
|
263
277
|
Resolve `{healthCheckFile}`: probe `{healthCheckProbeOrder}` in order. **HALT** if neither candidate exists — the health-check is the true terminal step; without it the workflow cannot complete honestly:
|
|
264
278
|
|
|
@@ -271,6 +285,12 @@ test-skill delegates its terminal step to the shared health-check. Install
|
|
|
271
285
|
the SKF module or run from a development checkout with src/ present.
|
|
272
286
|
```
|
|
273
287
|
|
|
288
|
+
**Headless envelope (if `{headless_mode}`):** emit to **stderr** before halting:
|
|
289
|
+
|
|
290
|
+
```
|
|
291
|
+
SKF_TEST_RESULT_JSON: {"status":"error","skill_name":"{skill_name}","verdict":null,"score":null,"threshold":null,"report_path":"{outputFile}","next_workflow":null,"exit_code":1,"halt_reason":"health-check-missing"}
|
|
292
|
+
```
|
|
293
|
+
|
|
274
294
|
Before displaying the menu, write the dispatch decision into the output report frontmatter (so the artifact records whether the health-check ran):
|
|
275
295
|
|
|
276
296
|
- `health_check_dispatched: true` — when the C menu choice will be taken (headless, or user will select C)
|
|
@@ -280,5 +300,5 @@ Also mirror the boolean into the `healthCheckDispatched` field of the result con
|
|
|
280
300
|
|
|
281
301
|
Display: "**Test complete.** [C] Finish"
|
|
282
302
|
|
|
283
|
-
On [C] (or auto-proceed in `{headless_mode}` — log: "headless: auto-continue past report menu"): set `health_check_dispatched: true` in frontmatter, then load and execute `{nextStepFile}` (the local health-check dispatcher). The test report document at `{outputFile}` contains the full analysis: Test Summary, Coverage Analysis, Coherence Analysis, Completeness Score, and Gap Report.
|
|
303
|
+
On [C] (or auto-proceed in `{headless_mode}` — log: "headless: auto-continue past report menu"): set `health_check_dispatched: true` in frontmatter, then load and execute `{nextStepFile}` (the local health-check dispatcher). The test report document at `{outputFile}` contains the full analysis: Test Summary, Coverage Analysis, Coherence Analysis, Completeness Score, and Gap Report. In `{headless_mode}`, once the dispatched health-check completes, the workflow makes its terminal process-exit with `{headless_exit_code}` (determined in §6b) — this is the single terminal exit for the headless happy path.
|
|
284
304
|
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
nextStepFile: 'report.md'
|
|
3
3
|
outputFile: '{forge_version}/test-report-{skill_name}-{run_id}.md'
|
|
4
|
-
scoringRulesFile: '
|
|
4
|
+
scoringRulesFile: '{scoringRulesPath}'
|
|
5
5
|
sourceAccessProtocol: 'references/source-access-protocol.md'
|
|
6
6
|
scoringScript: 'scripts/compute-score.py'
|
|
7
7
|
---
|
|
@@ -85,11 +85,13 @@ Build a JSON object from the data gathered in steps 1-2:
|
|
|
85
85
|
"coherence": "{combined_coherence_percentage or null if naive mode}",
|
|
86
86
|
"externalValidation": "{external_validation_score or null if N/A}"
|
|
87
87
|
},
|
|
88
|
-
"threshold": "{effective_threshold from §1 — CLI --threshold wins, then pipeline default, then workflow.default_threshold scalar, then 80}"
|
|
88
|
+
"threshold": "{effective_threshold from §1 — CLI --threshold wins, then pipeline default, then workflow.default_threshold scalar, then 80}",
|
|
89
|
+
"analysisConfidence": "{resolved analysis confidence — 'degraded' when python3/frontmatter validator missing; else full/provenance-map/metadata-only/remote-only/docs-only; omit if unknown}",
|
|
90
|
+
"toolingStatus": "{a missing-helper marker like 'python3-missing' or 'frontmatter-validator-missing' when a helper is unavailable, else omit}"
|
|
89
91
|
}
|
|
90
92
|
```
|
|
91
93
|
|
|
92
|
-
**Important:** Score values must be numbers (not strings). Use `null` (not `"N/A"`) for categories that were not scored. Read `metadata.json.skill_type` from `{resolved_skill_package}/metadata.json`; if the value is `"stack"`, set `stackSkill: true` and pass `null` for `signatureAccuracy` and `typeCoverage` (the categories will be redistributed per `{scoringRulesFile}` Stack Skills rule). Likewise read `metadata.json.scope_type`; if the value is `"reference-app"`, set `referenceApp: true` and pass `null` for `signatureAccuracy` and `typeCoverage` (redistributed per `{scoringRulesFile}` Reference-App rule — a reference app documents wiring patterns, not library export signatures).
|
|
94
|
+
**Important:** Score values must be numbers (not strings). Use `null` (not `"N/A"`) for categories that were not scored. `analysisConfidence` and `toolingStatus` are optional strings — pass them so the script can apply the post-score tooling cap (§3d) deterministically; the script fires Cap 1 when `analysisConfidence == "degraded"` or `toolingStatus` contains `missing`. Read `metadata.json.skill_type` from `{resolved_skill_package}/metadata.json`; if the value is `"stack"`, set `stackSkill: true` and pass `null` for `signatureAccuracy` and `typeCoverage` (the categories will be redistributed per `{scoringRulesFile}` Stack Skills rule). Likewise read `metadata.json.scope_type`; if the value is `"reference-app"`, set `referenceApp: true` and pass `null` for `signatureAccuracy` and `typeCoverage` (redistributed per `{scoringRulesFile}` Reference-App rule — a reference app documents wiring patterns, not library export signatures).
|
|
93
95
|
|
|
94
96
|
#### 3b. Run the Scoring Script
|
|
95
97
|
|
|
@@ -110,75 +112,54 @@ Parse the JSON output. The script returns:
|
|
|
110
112
|
- `skipReasons` — why each category was skipped
|
|
111
113
|
- `weightSum` — sum of final weights (should be ~100)
|
|
112
114
|
- `inconclusiveReasons` — only present when `result == "INCONCLUSIVE"`; explains which floor clause tripped
|
|
115
|
+
- **Verdict-override group** — present as an atomic set of four keys only when a post-score cap or the threshold fallback engaged (§3d/§4b are applied by the script, not re-derived here):
|
|
116
|
+
- `effectiveResult` — the **final verdict** after caps + fallback (`"PASS"` / `"FAIL"`); read this as the outcome for §4–§8
|
|
117
|
+
- `capReason` — string describing the cap(s) that fired, or `null`
|
|
118
|
+
- `thresholdFallback` — `true` when the FAIL→PASS-at-80-floor fallback fired
|
|
119
|
+
- `originalThreshold` — the pre-fallback threshold when `thresholdFallback` is `true`, else `null`
|
|
120
|
+
- When the group is **absent**, no cap or fallback engaged and `result` is the final verdict.
|
|
113
121
|
|
|
114
|
-
Use these values for Section 4 (pass/fail/inconclusive) and Section 6 (output formatting). **
|
|
122
|
+
Use these values for Section 4 (pass/fail/inconclusive) and Section 6 (output formatting). **The script owns the minimum-evidence floor, both post-score caps, and the threshold fallback — everywhere below, read its emitted fields (`result`, `effectiveResult`, `capReason`, `thresholdFallback`) and never recompute a verdict, cap, or threshold decision.** The final verdict is `effectiveResult` when the override group is present, otherwise `result` (INCONCLUSIVE is never overridden).
|
|
115
123
|
|
|
116
124
|
#### 3c. Fallback (if script execution fails)
|
|
117
125
|
|
|
118
|
-
If the script is unavailable or
|
|
126
|
+
If the script is unavailable or errors, redistribute each skipped category's weight (the `null`-scored categories in the §3a JSON — naive mode already zeroes coherence, and Quick-tier/docsOnly/state2/stackSkill/referenceApp already null out Signature Accuracy + Type Coverage) proportionally across the active categories, then report `total = Σ(weight/100 × category_score)` using the detected mode's weight table in `{scoringRulesFile}`. Report: "**Note:** Scoring script unavailable — calculated manually per scoring-rules.md."
|
|
119
127
|
|
|
120
|
-
|
|
121
|
-
2. Determine skip conditions: Quick tier/docsOnly/state2/stackSkill/referenceApp skip Signature Accuracy + Type Coverage; naive mode coherence is already 0; null external validation means skip it
|
|
122
|
-
3. For each skipped category, set its weight to 0
|
|
123
|
-
4. Compute sum of active category weights
|
|
124
|
-
5. For each active category: `new_weight = old_weight / sum_active * 100`
|
|
125
|
-
6. `weighted_score = new_weight / 100 * category_score`
|
|
126
|
-
7. `total = sum of all weighted_scores`
|
|
128
|
+
### 3d. Read Post-Score Caps (applied by the script)
|
|
127
129
|
|
|
128
|
-
|
|
129
|
-
|
|
130
|
-
### 3d. Apply Post-Score Caps (BEFORE PASS/FAIL determination)
|
|
131
|
-
|
|
132
|
-
The scoring script returns `totalScore` and a preliminary `result`. Two caps may override the script's PASS into FAIL — applied in this order, and ONLY if the script did NOT return INCONCLUSIVE (floor always takes precedence):
|
|
133
|
-
|
|
134
|
-
**Cap 1 — Tooling degraded:** if `analysisConfidence == "degraded"` or `toolingStatus` indicates a missing helper (`python3-missing`, `frontmatter-validator-missing`), set `effective_score = min(totalScore, threshold - 1)` and record `scoring_notes: tooling degraded — capped below threshold until helper restored`.
|
|
135
|
-
|
|
136
|
-
**Cap 2 — Docs-only without external validators:** if `docsOnly == true` AND `externalValidation` score was null (neither skill-check nor tessl ran), set `effective_score = min(effective_score, threshold - 1)` and record `scoring_notes: docs-only without external validators — capped below threshold`.
|
|
137
|
-
|
|
138
|
-
If either cap fired and `result` was `PASS`, change it to `FAIL`. If `result` was already `INCONCLUSIVE`, leave it — the evidence-floor verdict is never overridden by a cap.
|
|
130
|
+
The script applies two post-score caps — **Cap 1** (tooling degraded) and **Cap 2** (docs-only with no external validators) — and returns the settled outcome as `capReason` + `effectiveResult` (§3b). Read those fields; never recompute a cap. If `capReason` is non-null, record `scoring_notes: {capReason}` in the report. A fired cap forces the script's `PASS` into `FAIL`, never touches an INCONCLUSIVE verdict, and may still be re-flipped to PASS by the threshold fallback (§4b) — `effectiveResult` reflects that settled outcome. (Both caps exist because a degraded-tooling or docs-only-without-validators run has too thin an evidence base to trust a PASS; §3a passes the fields Cap 1 reads.)
|
|
139
131
|
|
|
140
132
|
### 4. Determine Result (PASS / FAIL / INCONCLUSIVE)
|
|
141
133
|
|
|
142
|
-
The scoring script enforces the minimum-evidence floor BEFORE comparing score vs threshold.
|
|
134
|
+
The scoring script enforces the minimum-evidence floor BEFORE comparing score vs threshold, then applies the post-score caps (§3d) and threshold fallback (§4b). The **settled verdict** is `effectiveResult` when the override group is present, otherwise `result`:
|
|
143
135
|
|
|
144
136
|
```
|
|
145
|
-
IF result == "INCONCLUSIVE" — minimum-evidence floor tripped; not PASS, not FAIL
|
|
146
|
-
|
|
147
|
-
IF score < threshold → FAIL
|
|
137
|
+
IF result == "INCONCLUSIVE" — minimum-evidence floor tripped; not PASS, not FAIL (never overridden)
|
|
138
|
+
ELSE settled verdict = effectiveResult if the override group is present, else result (PASS or FAIL)
|
|
148
139
|
```
|
|
149
140
|
|
|
150
141
|
**INCONCLUSIVE floor clauses** (see `{scoringRulesFile}`):
|
|
151
142
|
- `active_categories < 2` (after all redistribution), OR
|
|
152
143
|
- `tier == "Quick"` AND Export Coverage is the sole scoring contributor
|
|
153
144
|
|
|
154
|
-
**Tooling-degraded cap:** If the `analysisConfidence` in output frontmatter is `degraded` (python3 missing, frontmatter validator missing, or other degraded state flagged in step 1), the step MUST cap the score at `threshold - 1` BEFORE the PASS/FAIL comparison. This forces a deterministic FAIL until tooling is restored. Do NOT override an INCONCLUSIVE result with the cap — INCONCLUSIVE remains the verdict.
|
|
155
|
-
|
|
156
145
|
### 4b. Threshold Fallback and Evidence Report
|
|
157
146
|
|
|
158
|
-
After §4 determines the result but before §5 recommends the next workflow,
|
|
159
|
-
|
|
160
|
-
**Fallback trigger conditions:**
|
|
147
|
+
After §4 determines the result but before §5 recommends the next workflow, the script's threshold fallback may already have converted a FAIL into a PASS at the 80% floor. This step documents that quality compromise in an evidence report.
|
|
161
148
|
|
|
162
|
-
|
|
163
|
-
IF result == "FAIL"
|
|
164
|
-
AND totalScore >= 80
|
|
165
|
-
AND effective_threshold > 80
|
|
166
|
-
THEN
|
|
167
|
-
→ THRESHOLD FALLBACK triggered
|
|
168
|
-
```
|
|
149
|
+
The rule the script encodes (documented for interpretation): when `result == "FAIL"` AND `totalScore >= 80` AND `effective_threshold > 80`, the script overrides `result` to `"PASS"` at the 80% floor; an `INCONCLUSIVE` verdict is never overridden. Read the script's fields:
|
|
169
150
|
|
|
170
|
-
|
|
171
|
-
- `
|
|
172
|
-
- `totalScore < 80` — that is a genuine FAIL (floor not met)
|
|
173
|
-
- `effective_threshold == 80` — there is nothing to fall back from
|
|
151
|
+
- `thresholdFallback == true` → the fallback fired; `effectiveResult` is `"PASS"` and `originalThreshold` holds the pre-fallback threshold.
|
|
152
|
+
- `thresholdFallback` absent or `false` → no fallback; the settled verdict from §4 stands.
|
|
174
153
|
|
|
175
|
-
**When
|
|
154
|
+
**When `thresholdFallback` is true:**
|
|
176
155
|
|
|
177
|
-
1. Record `threshold_fallback: true`, `original_threshold: {
|
|
178
|
-
2.
|
|
156
|
+
1. Record `threshold_fallback: true`, `original_threshold: {originalThreshold}`, `fallback_threshold: 80` in workflow context.
|
|
157
|
+
2. Use `effectiveResult` (`"PASS"`) as the settled verdict — do not recompute it.
|
|
179
158
|
3. Set `effective_threshold = 80` for use by §5/§6/§7/§8.
|
|
180
159
|
4. Generate the evidence report (§4b.1 below).
|
|
181
160
|
|
|
161
|
+
The cap↔fallback interaction (a cap forcing FAIL that the fallback then re-flips to PASS at the 80 floor) is resolved inside the script — `effectiveResult` reflects the settled outcome. The `{totalScore}` used in the report below is the script's raw `totalScore`.
|
|
162
|
+
|
|
182
163
|
#### 4b.1 Generate Evidence Report
|
|
183
164
|
|
|
184
165
|
Write the evidence report to `{forge_version}/evidence-report-fallback.md`. The report documents the quality compromise for audit purposes.
|
|
@@ -234,7 +215,7 @@ Record `evidence_report_path: '{forge_version}/evidence-report-fallback.md'` in
|
|
|
234
215
|
|
|
235
216
|
### 5. Determine Next Workflow Recommendation
|
|
236
217
|
|
|
237
|
-
Based on
|
|
218
|
+
Based on the **settled verdict** (§4 — `effectiveResult` when the override group is present, else `result`; INCONCLUSIVE is never overridden):
|
|
238
219
|
|
|
239
220
|
**IF PASS:**
|
|
240
221
|
- `nextWorkflow: 'export-skill'` — skill is ready for export
|
|
@@ -301,13 +282,13 @@ If `analysis_confidence` is not `full`, append a degradation notice. **The notic
|
|
|
301
282
|
|
|
302
283
|
**Resolved via:** {analysis_confidence} {confidence breakdown if provenance-map, e.g., "(T1 AST-verified at compilation time)" or "(12 T1, 3 T1-low)"}
|
|
303
284
|
**Impact:** {describe limitation — e.g., "Signature checks limited to name-matching. Source file:line citations from provenance-map, not live AST." — or "Provenance data is at highest confidence; no limitation." for all-T1 provenance-map}
|
|
304
|
-
**Recommendation:** {confidence-dependent — see {sourceAccessProtocol} degradation notice rules. Do
|
|
285
|
+
**Recommendation:** {confidence-dependent — see {sourceAccessProtocol} degradation notice rules. Do not recommend local clone when provenance-map entries are already T1.}
|
|
305
286
|
```
|
|
306
287
|
|
|
307
288
|
### 7. Update Output Frontmatter
|
|
308
289
|
|
|
309
290
|
Update `{outputFile}` frontmatter:
|
|
310
|
-
- `testResult: '{pass|pass-with-drift|fail|inconclusive}'` (lowercase; mirrors
|
|
291
|
+
- `testResult: '{pass|pass-with-drift|fail|inconclusive}'` (lowercase; mirrors the **settled verdict** — `effectiveResult` when the override group is present, else `result` — with `pass-with-drift` substituted for `pass` when `allow_workspace_drift` was set and drift was observed — see §5 drift override)
|
|
311
292
|
- `score: '{total}%'`
|
|
312
293
|
- `threshold: '{threshold}%'`
|
|
313
294
|
- `thresholdSource: '{threshold_source}'`
|
|
@@ -318,24 +299,11 @@ Update `{outputFile}` frontmatter:
|
|
|
318
299
|
|
|
319
300
|
### 8. Report Score
|
|
320
301
|
|
|
321
|
-
|
|
322
|
-
|
|
323
|
-
**{skill_name}:** **{total}%** — **{PASS|FAIL}**
|
|
302
|
+
Report the completeness score to the user: the total percentage and PASS/FAIL verdict, the per-category weighted breakdown (already appended to the report in §6), the threshold, and the recommended next workflow (export-skill on pass, update-skill on fail). When `threshold_fallback` is true, include the fallback notice:
|
|
324
303
|
|
|
325
|
-
| Category | Score | Weighted |
|
|
326
|
-
|----------|-------|----------|
|
|
327
|
-
| Export Coverage | {N}% | {WS}% |
|
|
328
|
-
| Signature Accuracy | {N}% | {WS}% |
|
|
329
|
-
| Type Coverage | {N}% | {WS}% |
|
|
330
|
-
| Coherence | {N}% | {WS}% |
|
|
331
|
-
| External Validation | {N}% | {WS}% |
|
|
332
|
-
|
|
333
|
-
**Threshold:** {threshold}%
|
|
334
|
-
{If threshold_fallback is true:}
|
|
335
304
|
**Threshold fallback:** scored {totalScore}% against {original_threshold}% target — accepted at 80% floor. Evidence report: {evidence_report_path}
|
|
336
|
-
**Recommendation:** {export-skill if pass | update-skill if fail}
|
|
337
305
|
|
|
338
|
-
|
|
306
|
+
Then proceed to the gap report.
|
|
339
307
|
|
|
340
308
|
Update stepsCompleted, then load and execute {nextStepFile}.
|
|
341
309
|
|