bmad-module-skill-forge 1.4.1 → 1.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/marketplace.json +1 -1
- package/README.md +0 -8
- package/docs/_data/pinned.yaml +1 -1
- package/docs/_internal/STABILITY.md +1 -1
- package/docs/architecture.md +2 -2
- package/docs/examples.md +1 -1
- package/docs/workflows.md +5 -5
- package/package.json +2 -2
- package/src/README.md +1 -1
- package/src/knowledge/ccc-bridge.md +12 -12
- package/src/knowledge/qmd-registry.md +5 -5
- package/src/shared/health-check.md +5 -5
- package/src/shared/references/description-guard-protocol.md +100 -0
- package/src/shared/references/output-contract-schema.md +1 -1
- package/src/shared/scripts/schemas/skf-brief-result-envelope.v1.json +6 -6
- package/src/shared/scripts/schemas/skf-setup-result-envelope.v1.json +9 -3
- package/src/shared/scripts/schemas/skf-update-result-envelope.v1.json +149 -0
- package/src/shared/scripts/schemas/skill-brief.v1.json +22 -2
- package/src/shared/scripts/skf-build-change-manifest.py +420 -0
- package/src/shared/scripts/skf-check-workspace-drift.py +321 -0
- package/src/shared/scripts/skf-compare-file-hashes.py +357 -0
- package/src/shared/scripts/skf-description-guard.py +359 -0
- package/src/shared/scripts/skf-detect-language.py +2 -2
- package/src/shared/scripts/skf-detect-scripts-assets.py +613 -0
- package/src/shared/scripts/skf-detect-tools.py +97 -8
- package/src/shared/scripts/skf-detect-workspaces.py +1 -1
- package/src/shared/scripts/skf-disqualify-candidates.py +576 -0
- package/src/shared/scripts/skf-emit-brief-result-envelope.py +3 -3
- package/src/shared/scripts/skf-emit-result-envelope.py +93 -9
- package/src/shared/scripts/skf-enumerate-stack-skills.py +514 -0
- package/src/shared/scripts/skf-forge-tier-rw.py +11 -11
- package/src/shared/scripts/skf-hash-content.py +284 -0
- package/src/shared/scripts/skf-load-provenance.py +295 -0
- package/src/shared/scripts/skf-merge-ccc-exclusions.py +34 -4
- package/src/shared/scripts/skf-pair-intersect.py +250 -0
- package/src/shared/scripts/skf-provenance-gap-dispatch.py +433 -0
- package/src/shared/scripts/skf-qmd-classify-collections.py +42 -8
- package/src/shared/scripts/skf-recommend-scope-type.py +2 -2
- package/src/shared/scripts/skf-render-quick-metadata.py +2 -2
- package/src/shared/scripts/skf-resolve-authoritative-files.py +504 -0
- package/src/shared/scripts/skf-scan-manifests.py +738 -0
- package/src/shared/scripts/skf-scan-skill-md-structure.py +375 -0
- package/src/shared/scripts/skf-update-active-symlink.py +315 -0
- package/src/shared/scripts/skf-validate-brief-inputs.py +5 -5
- package/src/shared/scripts/skf-validate-brief-schema.py +348 -0
- package/src/shared/scripts/skf-write-skill-brief.py +61 -5
- package/src/skf-analyze-source/SKILL.md +69 -16
- package/src/skf-analyze-source/assets/skill-brief-schema.md +1 -1
- package/src/skf-analyze-source/customize.toml +55 -0
- package/src/skf-analyze-source/{steps-c/step-01b-continue.md → references/continue.md} +12 -12
- package/src/skf-analyze-source/{steps-c/step-06-generate-briefs.md → references/generate-briefs.md} +29 -10
- package/src/skf-analyze-source/{steps-c/step-07-health-check.md → references/health-check.md} +5 -3
- package/src/skf-analyze-source/{steps-c/step-03-identify-units.md → references/identify-units.md} +44 -14
- package/src/skf-analyze-source/{steps-c/step-01-init.md → references/init.md} +27 -6
- package/src/skf-analyze-source/{steps-c/step-04-map-and-detect.md → references/map-and-detect.md} +52 -25
- package/src/skf-analyze-source/{steps-c/step-05-recommend.md → references/recommend.md} +15 -8
- package/src/skf-analyze-source/{steps-c/step-02-scan-project.md → references/scan-project.md} +12 -10
- package/src/skf-analyze-source/references/unit-detection-heuristics.md +2 -0
- package/src/skf-analyze-source/templates/analysis-report-template.md +6 -6
- package/src/skf-audit-skill/SKILL.md +66 -15
- package/src/skf-audit-skill/assets/drift-report-template.md +6 -5
- package/src/skf-audit-skill/customize.toml +49 -0
- package/src/skf-audit-skill/{steps-c/step-07-health-check.md → references/health-check.md} +5 -3
- package/src/skf-audit-skill/{steps-c/step-01-init.md → references/init.md} +57 -19
- package/src/skf-audit-skill/{steps-c/step-02-re-index.md → references/re-index.md} +11 -11
- package/src/skf-audit-skill/{steps-c/step-06-report.md → references/report.md} +25 -7
- package/src/skf-audit-skill/{steps-c/step-04-semantic-diff.md → references/semantic-diff.md} +6 -6
- package/src/skf-audit-skill/{steps-c/step-05-severity-classify.md → references/severity-classify.md} +5 -5
- package/src/skf-audit-skill/references/severity-rules.md +2 -0
- package/src/skf-audit-skill/{steps-c/step-03-structural-diff.md → references/structural-diff.md} +29 -17
- package/src/skf-brief-skill/SKILL.md +48 -18
- package/src/skf-brief-skill/assets/description-voice-examples.md +1 -1
- package/src/skf-brief-skill/assets/skill-brief-schema.md +32 -2
- package/src/skf-brief-skill/customize.toml +44 -0
- package/src/skf-brief-skill/{steps-c/step-02-analyze-target.md → references/analyze-target.md} +5 -5
- package/src/skf-brief-skill/{steps-c/step-04-confirm-brief.md → references/confirm-brief.md} +13 -10
- package/src/skf-brief-skill/references/draft-checkpoint.md +4 -4
- package/src/skf-brief-skill/{steps-c/step-01-gather-intent.md → references/gather-intent.md} +24 -24
- package/src/skf-brief-skill/references/headless-args.md +9 -9
- package/src/skf-brief-skill/references/headless-source-authority-detection.md +1 -1
- package/src/skf-brief-skill/{steps-c/step-06-health-check.md → references/health-check.md} +4 -2
- package/src/skf-brief-skill/references/portfolio-similarity-check.md +2 -2
- package/src/skf-brief-skill/references/qmd-collection-registration.md +1 -1
- package/src/skf-brief-skill/{steps-c/step-03-scope-definition.md → references/scope-definition.md} +19 -16
- package/src/skf-brief-skill/references/version-resolution.md +2 -2
- package/src/skf-brief-skill/{steps-c/step-05-write-brief.md → references/write-brief.md} +11 -10
- package/src/skf-create-skill/SKILL.md +25 -19
- package/src/skf-create-skill/assets/compile-assembly-rules.md +3 -3
- package/src/skf-create-skill/assets/skill-sections.md +5 -5
- package/src/skf-create-skill/assets/tessl-dismissal-rules.md +11 -11
- package/src/skf-create-skill/customize.toml +35 -0
- package/src/skf-create-skill/references/authoritative-files-protocol.md +142 -0
- package/src/skf-create-skill/{steps-c/step-05-compile.md → references/compile.md} +16 -16
- package/src/skf-create-skill/{steps-c/step-03d-component-extraction.md → references/component-extraction.md} +16 -16
- package/src/skf-create-skill/{steps-c/step-02-ecosystem-check.md → references/ecosystem-check.md} +4 -4
- package/src/skf-create-skill/{steps-c/step-04-enrich.md → references/enrich.md} +3 -3
- package/src/skf-create-skill/{steps-c/step-03-extract.md → references/extract.md} +52 -120
- package/src/skf-create-skill/references/extraction-patterns.md +8 -8
- package/src/skf-create-skill/{steps-c/step-07-generate-artifacts.md → references/generate-artifacts.md} +11 -11
- package/src/skf-create-skill/{steps-c/step-09-health-check.md → references/health-check.md} +4 -2
- package/src/skf-create-skill/{steps-c/step-01-load-brief.md → references/load-brief.md} +40 -32
- package/src/skf-create-skill/{steps-c/step-08-report.md → references/report.md} +8 -8
- package/src/skf-create-skill/references/source-resolution-protocols.md +6 -6
- package/src/skf-create-skill/{steps-c/sub/step-02b-ccc-discover.md → references/sub/ccc-discover.md} +5 -5
- package/src/skf-create-skill/{steps-c/sub/step-03c-fetch-docs.md → references/sub/fetch-docs.md} +28 -12
- package/src/skf-create-skill/{steps-c/sub/step-03b-fetch-temporal.md → references/sub/fetch-temporal.md} +81 -50
- package/src/skf-create-skill/{steps-c/step-06-validate.md → references/validate.md} +46 -26
- package/src/skf-create-stack-skill/SKILL.md +76 -17
- package/src/skf-create-stack-skill/assets/provenance-map-schema.md +102 -0
- package/src/skf-create-stack-skill/assets/stack-skill-template.md +1 -1
- package/src/skf-create-stack-skill/customize.toml +46 -0
- package/src/skf-create-stack-skill/{steps-c/step-06-compile-stack.md → references/compile-stack.md} +6 -5
- package/src/skf-create-stack-skill/references/compose-mode-rules.md +3 -1
- package/src/skf-create-stack-skill/{steps-c/step-05-detect-integrations.md → references/detect-integrations.md} +32 -11
- package/src/skf-create-stack-skill/{steps-c/step-02-detect-manifests.md → references/detect-manifests.md} +28 -28
- package/src/skf-create-stack-skill/{steps-c/step-07-generate-output.md → references/generate-output.md} +11 -85
- package/src/skf-create-stack-skill/{steps-c/step-10-health-check.md → references/health-check.md} +4 -2
- package/src/skf-create-stack-skill/{steps-c/step-01-init.md → references/init.md} +70 -4
- package/src/skf-create-stack-skill/references/integration-patterns.md +2 -0
- package/src/skf-create-stack-skill/references/manifest-patterns.md +2 -0
- package/src/skf-create-stack-skill/{steps-c/step-04-parallel-extract.md → references/parallel-extract.md} +40 -17
- package/src/skf-create-stack-skill/{steps-c/step-03-rank-and-confirm.md → references/rank-and-confirm.md} +5 -4
- package/src/skf-create-stack-skill/{steps-c/step-09-report.md → references/report.md} +3 -3
- package/src/skf-create-stack-skill/{steps-c/step-08-validate.md → references/validate.md} +6 -6
- package/src/skf-drop-skill/SKILL.md +84 -12
- package/src/skf-drop-skill/customize.toml +57 -0
- package/src/skf-drop-skill/{steps-c/step-02-execute.md → references/execute.md} +49 -34
- package/src/skf-drop-skill/{steps-c/step-04-health-check.md → references/health-check.md} +6 -4
- package/src/skf-drop-skill/{steps-c/step-03-report.md → references/report.md} +13 -5
- package/src/skf-drop-skill/{steps-c/step-01-select.md → references/select.md} +63 -15
- package/src/skf-export-skill/SKILL.md +85 -16
- package/src/skf-export-skill/assets/managed-section-format.md +3 -3
- package/src/skf-export-skill/assets/snippet-format.md +2 -2
- package/src/skf-export-skill/customize.toml +49 -0
- package/src/skf-export-skill/{steps-c/step-03-generate-snippet.md → references/generate-snippet.md} +11 -11
- package/src/skf-export-skill/{steps-c/step-07-health-check.md → references/health-check.md} +6 -4
- package/src/skf-export-skill/{steps-c/step-01-load-skill.md → references/load-skill.md} +11 -49
- package/src/skf-export-skill/references/manifest-rebuild.md +68 -0
- package/src/skf-export-skill/references/multi-skill-mode.md +38 -0
- package/src/skf-export-skill/references/orphan-context-detection.md +75 -0
- package/src/skf-export-skill/references/orphan-row-detection.md +102 -0
- package/src/skf-export-skill/{steps-c/step-02-package.md → references/package.md} +7 -7
- package/src/skf-export-skill/references/preflight-snippet-root-probe.md +74 -0
- package/src/skf-export-skill/{steps-c/step-06-summary.md → references/summary.md} +20 -8
- package/src/skf-export-skill/{steps-c/step-05-token-report.md → references/token-report.md} +5 -5
- package/src/skf-export-skill/{steps-c/step-04-update-context.md → references/update-context.md} +109 -113
- package/src/skf-quick-skill/SKILL.md +63 -123
- package/src/skf-quick-skill/customize.toml +44 -0
- package/src/skf-quick-skill/references/batch-mode.md +102 -0
- package/src/skf-quick-skill/{steps-c/step-04-compile.md → references/compile.md} +10 -10
- package/src/skf-quick-skill/{steps-c/step-02-ecosystem-check.md → references/ecosystem-check.md} +5 -5
- package/src/skf-quick-skill/{steps-c/step-06-finalize.md → references/finalize.md} +7 -7
- package/src/skf-quick-skill/{steps-c/step-07-health-check.md → references/health-check.md} +3 -3
- package/src/skf-quick-skill/{steps-c/step-03-quick-extract.md → references/quick-extract.md} +5 -5
- package/src/skf-quick-skill/references/registry-resolution.md +2 -0
- package/src/skf-quick-skill/{steps-c/step-01-resolve-target.md → references/resolve-target.md} +14 -10
- package/src/skf-quick-skill/{steps-c/step-05-write-and-validate.md → references/write-and-validate.md} +4 -4
- package/src/skf-refine-architecture/SKILL.md +86 -16
- package/src/skf-refine-architecture/customize.toml +49 -0
- package/src/skf-refine-architecture/{steps-c/step-05-compile.md → references/compile.md} +4 -4
- package/src/skf-refine-architecture/{steps-c/step-02-gap-analysis.md → references/gap-analysis.md} +5 -5
- package/src/skf-refine-architecture/{steps-c/step-07-health-check.md → references/health-check.md} +6 -4
- package/src/skf-refine-architecture/{steps-c/step-04-improvements.md → references/improvements.md} +5 -5
- package/src/skf-refine-architecture/references/init.md +144 -0
- package/src/skf-refine-architecture/{steps-c/step-03-issue-detection.md → references/issue-detection.md} +5 -5
- package/src/skf-refine-architecture/references/refinement-rules.md +2 -0
- package/src/skf-refine-architecture/{steps-c/step-06-report.md → references/report.md} +14 -5
- package/src/skf-rename-skill/SKILL.md +82 -12
- package/src/skf-rename-skill/customize.toml +52 -0
- package/src/skf-rename-skill/{steps-c/step-02-execute.md → references/execute.md} +94 -99
- package/src/skf-rename-skill/references/health-check.md +30 -0
- package/src/skf-rename-skill/references/rebuild-context.md +110 -0
- package/src/skf-rename-skill/{steps-c/step-03-report.md → references/report.md} +13 -5
- package/src/skf-rename-skill/{steps-c/step-01-select.md → references/select.md} +59 -20
- package/src/skf-setup/SKILL.md +35 -35
- package/src/skf-setup/customize.toml +33 -0
- package/src/skf-setup/{steps-c/step-03-auto-index.md → references/auto-index.md} +10 -18
- package/src/skf-setup/{steps-c/step-01b-ccc-index.md → references/ccc-index.md} +16 -31
- package/src/skf-setup/{steps-c/step-01-detect-and-tier.md → references/detect-and-tier.md} +23 -10
- package/src/skf-setup/{steps-c/step-05-health-check.md → references/health-check.md} +1 -1
- package/src/skf-setup/{steps-c/step-04-report.md → references/report.md} +21 -19
- package/src/skf-setup/references/tier-rules.md +1 -1
- package/src/skf-setup/{steps-c/step-02-write-config.md → references/write-config.md} +14 -13
- package/src/skf-test-skill/SKILL.md +77 -15
- package/src/skf-test-skill/customize.toml +54 -0
- package/src/skf-test-skill/{steps-c/step-04-coherence-check.md → references/coherence-check.md} +28 -71
- package/src/skf-test-skill/{steps-c/step-03-coverage-check.md → references/coverage-check.md} +16 -39
- package/src/skf-test-skill/{steps-c/step-02-detect-mode.md → references/detect-mode.md} +4 -26
- package/src/skf-test-skill/{steps-c/step-04b-external-validators.md → references/external-validators.md} +6 -28
- package/src/skf-test-skill/references/health-check.md +14 -0
- package/src/skf-test-skill/{steps-c/step-01-init.md → references/init.md} +21 -37
- package/src/skf-test-skill/references/migration-section-rules.md +4 -2
- package/src/skf-test-skill/{steps-c/step-06-report.md → references/report.md} +41 -51
- package/src/skf-test-skill/{steps-c/step-05-score.md → references/score.md} +26 -43
- package/src/skf-test-skill/references/scoring-rules.md +8 -6
- package/src/skf-test-skill/references/source-access-protocol.md +7 -5
- package/src/skf-test-skill/scripts/compute-score.py +88 -17
- package/src/skf-test-skill/templates/test-report-template.md +13 -13
- package/src/skf-update-skill/SKILL.md +23 -14
- package/src/skf-update-skill/customize.toml +44 -0
- package/src/skf-update-skill/{steps-c/step-02-detect-changes.md → references/detect-changes.md} +176 -68
- package/src/skf-update-skill/references/health-check.md +32 -0
- package/src/skf-update-skill/{steps-c/step-01-init.md → references/init.md} +56 -8
- package/src/skf-update-skill/references/manual-section-rules.md +4 -0
- package/src/skf-update-skill/references/merge-conflict-rules.md +4 -0
- package/src/skf-update-skill/{steps-c/step-04-merge.md → references/merge.md} +8 -8
- package/src/skf-update-skill/{steps-c/step-03-re-extract.md → references/re-extract.md} +53 -44
- package/src/skf-update-skill/references/remote-source-resolution.md +5 -1
- package/src/skf-update-skill/references/report.md +214 -0
- package/src/skf-update-skill/{steps-c/step-05-validate.md → references/validate.md} +5 -5
- package/src/skf-update-skill/{steps-c/step-06-write.md → references/write.md} +73 -51
- package/src/skf-verify-stack/SKILL.md +89 -17
- package/src/skf-verify-stack/assets/feasibility-report-template.md +4 -4
- package/src/skf-verify-stack/customize.toml +50 -0
- package/src/skf-verify-stack/references/coverage-patterns.md +2 -2
- package/src/skf-verify-stack/{steps-c/step-02-coverage.md → references/coverage.md} +7 -7
- package/src/skf-verify-stack/{steps-c/step-07-health-check.md → references/health-check.md} +7 -5
- package/src/skf-verify-stack/references/init.md +166 -0
- package/src/skf-verify-stack/references/integration-verification-rules.md +1 -1
- package/src/skf-verify-stack/{steps-c/step-03-integrations.md → references/integrations.md} +12 -12
- package/src/skf-verify-stack/{steps-c/step-06-report.md → references/report.md} +22 -41
- package/src/skf-verify-stack/{steps-c/step-04-requirements.md → references/requirements.md} +7 -7
- package/src/skf-verify-stack/{steps-c/step-05-synthesize.md → references/synthesize.md} +8 -8
- package/tools/validate-docs-drift.js +1 -1
- package/src/skf-refine-architecture/steps-c/step-01-init.md +0 -136
- package/src/skf-rename-skill/steps-c/step-04-health-check.md +0 -22
- package/src/skf-test-skill/steps-c/step-07-health-check.md +0 -25
- package/src/skf-update-skill/steps-c/step-07-report.md +0 -148
- package/src/skf-update-skill/steps-c/step-08-health-check.md +0 -22
- package/src/skf-verify-stack/steps-c/step-01-init.md +0 -178
package/src/skf-test-skill/{steps-c/step-04-coherence-check.md → references/coherence-check.md}
RENAMED
|
@@ -1,27 +1,20 @@
|
|
|
1
1
|
---
|
|
2
|
-
nextStepFile: '
|
|
2
|
+
nextStepFile: 'external-validators.md'
|
|
3
3
|
outputFile: '{forge_version}/test-report-{skill_name}-{run_id}.md'
|
|
4
4
|
outputFormatsFile: 'assets/output-section-formats.md'
|
|
5
5
|
scoringRulesFile: 'references/scoring-rules.md'
|
|
6
6
|
migrationSectionRules: 'references/migration-section-rules.md'
|
|
7
|
+
scanSkillMdStructureScript: '{project-root}/src/shared/scripts/skf-scan-skill-md-structure.py'
|
|
7
8
|
---
|
|
8
9
|
|
|
10
|
+
<!-- Config: communicate in {communication_language}. -->
|
|
11
|
+
|
|
9
12
|
# Step 4: Coherence Check
|
|
10
13
|
|
|
11
14
|
## STEP GOAL:
|
|
12
15
|
|
|
13
16
|
Validate internal consistency of the skill documentation. In contextual mode (stack skills): verify that all cross-references in SKILL.md point to real files, types match their declarations, and integration patterns are complete. In naive mode (individual skills): perform basic structural validation only.
|
|
14
17
|
|
|
15
|
-
## Rules
|
|
16
|
-
|
|
17
|
-
- Use subprocess optimization: grep for references, then per-reference deep validation
|
|
18
|
-
- For each reference in contextual mode, launch a subprocess to validate the target — do not shortcut
|
|
19
|
-
- Analysis depth is conditional on testMode (naive vs contextual)
|
|
20
|
-
|
|
21
|
-
## MANDATORY SEQUENCE
|
|
22
|
-
|
|
23
|
-
**CRITICAL:** Follow this sequence exactly. Do not skip, reorder, or improvise unless user explicitly requests a change.
|
|
24
|
-
|
|
25
18
|
### 1. Check Test Mode
|
|
26
19
|
|
|
27
20
|
Read `testMode` from `{outputFile}` frontmatter.
|
|
@@ -29,40 +22,32 @@ Read `testMode` from `{outputFile}` frontmatter.
|
|
|
29
22
|
**IF naive mode → Execute Naive Coherence (Section 2)**
|
|
30
23
|
**IF contextual mode → Execute Contextual Coherence (Sections 3-5)**
|
|
31
24
|
|
|
32
|
-
### 2. Naive Mode: Concrete Structural Validation
|
|
25
|
+
### 2. Naive Mode: Concrete Structural Validation
|
|
33
26
|
|
|
34
|
-
Perform the following explicit checks (no hand-waving —
|
|
27
|
+
Perform the following explicit checks (no hand-waving — most use a single deterministic script; severity assignments are binding; do not relax them).
|
|
35
28
|
|
|
36
|
-
**2.
|
|
37
|
-
- Description: `## Description` OR frontmatter `description` field — either satisfies
|
|
38
|
-
- Usage: `## Usage` OR `## Examples` OR `## Quick Start` OR `## Common Workflows`
|
|
39
|
-
- API surface: `## Exports` OR `## Key API Summary` OR `## API`
|
|
40
|
-
- **Zero matches across an entire synonym set → High severity** finding: `naive-coherence — missing required section: {section-set-name}`
|
|
29
|
+
**2.0 Run the structural scan.** Invoke `{scanSkillMdStructureScript}` twice and parse the JSON outputs. These results back §§2.1, 2.2, 2.3, and 2.6 — do not re-implement those checks with grep/sed/awk loops.
|
|
41
30
|
|
|
42
|
-
|
|
31
|
+
```bash
|
|
32
|
+
uv run {scanSkillMdStructureScript} scan {skill-md} --required-sections
|
|
33
|
+
uv run {scanSkillMdStructureScript} scan {skill-md}
|
|
34
|
+
```
|
|
43
35
|
|
|
44
|
-
|
|
36
|
+
The first call returns `{ description: {satisfied, matched_synonym, tried[]}, usage: {...}, api_surface: {...} }`. The second returns `{ unbalanced_fences, fence_count, bare_opening_fences[{line,text}], table_drift[{line,section,expected_cols,actual_cols,row}] }`. Hold both JSON blobs for the checks below.
|
|
45
37
|
|
|
46
|
-
**2.
|
|
38
|
+
**2.1 Required sections present.** Read the first JSON blob from §2.0. For each of the three families (`description`, `usage`, `api_surface`):
|
|
47
39
|
|
|
48
|
-
|
|
40
|
+
- `satisfied: true` → no finding.
|
|
41
|
+
- `satisfied: false` AND family is `description` AND the SKILL.md frontmatter has a non-empty `description` field → no finding (the frontmatter alternative satisfies the family per the original rule).
|
|
42
|
+
- Otherwise → **High severity** finding: `naive-coherence — missing required section: {family}` (the `tried[]` list from the JSON identifies which synonyms were checked: `Description`/`Overview`/`Purpose`/`Summary` for description; `Usage`/`Examples`/`How to use`/`Quickstart`/`Quick Start`/`Getting Started`/`Common Workflows` for usage; `API`/`API Surface`/`Exports`/`Public API`/`Interface`/`Reference`/`Key API Summary` for api_surface).
|
|
49
43
|
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
s = line.rstrip('\n')
|
|
54
|
-
if s.startswith('```'):
|
|
55
|
-
if not in_code:
|
|
56
|
-
if s == '```':
|
|
57
|
-
print(f'{i}: bare opening fence')
|
|
58
|
-
in_code = True
|
|
59
|
-
else:
|
|
60
|
-
in_code = False
|
|
61
|
-
```
|
|
44
|
+
The script matches case-insensitively and tolerates `##`/`###` heading levels. SKF-template skills' `## Quick Start`, `## Common Workflows`, and `## Key API Summary` are first-class synonyms baked into the script — they will surface with `satisfied: true` and the corresponding `matched_synonym` field.
|
|
45
|
+
|
|
46
|
+
**2.2 Code fence balance.** Read `unbalanced_fences` from the second JSON blob. **`true` → High severity** finding: `naive-coherence — unbalanced code fence (unclosed block)` (the JSON's `fence_count` may be cited in the detail).
|
|
62
47
|
|
|
63
|
-
**
|
|
48
|
+
**2.3 Language tags on opening fences.** Read `bare_opening_fences[]` from the second JSON blob. The script already runs the stateful open/close scan — closing fences are never reported. For each entry, emit a **Medium severity** finding: `naive-coherence — opening code fence at line {entry.line} missing language tag`.
|
|
64
49
|
|
|
65
|
-
**2.4 Exports cross-used in Usage section.** For each function name reported in the step
|
|
50
|
+
**2.4 Exports cross-used in Usage section.** For each function name reported in the step 3 subagent inventory (`exports[].name` where `kind == "function"` or `kind == "method"`):
|
|
66
51
|
- `grep -c "{export.name}" SKILL.md` restricted to the Usage section (find the `## Usage` anchor from §2.1 and the next `^## ` anchor; count within that span).
|
|
67
52
|
- **Zero occurrences → High severity** finding: `naive-coherence — exported {kind} \`{name}\` is not referenced in the Usage section`. This catches the "documented but unused" failure mode that trivially fails discovery testing.
|
|
68
53
|
|
|
@@ -70,20 +55,7 @@ for i, line in enumerate(open('SKILL.md'), 1):
|
|
|
70
55
|
- Description says async + example shows no `await` → **High severity** finding: `naive-coherence — \`{name}\` described as async but example lacks \`await\``
|
|
71
56
|
- Description says sync + example uses `await {name}` → **High severity** finding: `naive-coherence — \`{name}\` described as sync but example awaits it`
|
|
72
57
|
|
|
73
|
-
**2.6 Table syntax.** `
|
|
74
|
-
|
|
75
|
-
Recipe:
|
|
76
|
-
|
|
77
|
-
```bash
|
|
78
|
-
# Normalize `\|` to a placeholder, split on |, count, restore.
|
|
79
|
-
grep -nE '^\|.*\|$' SKILL.md \
|
|
80
|
-
| sed 's/\\|/\x00/g' \
|
|
81
|
-
| awk -F'|' '{print NR, NF-2}' # -2 drops the empty leading/trailing fields
|
|
82
|
-
```
|
|
83
|
-
|
|
84
|
-
Or equivalent: hand off to a proper markdown-table parser. A plain `split on |` WILL produce false "column drift" findings on any table whose cells contain union types.
|
|
85
|
-
|
|
86
|
-
**Column-count drift → Medium severity** finding: `naive-coherence — table row at line {N} has {X} columns; neighboring rows have {Y}`.
|
|
58
|
+
**2.6 Table syntax.** Read `table_drift[]` from the second JSON blob (§2.0). The script normalizes escaped pipes (`\|`, used inside TypeScript union types such as `string \| undefined`) before splitting and compares each row against its header's column count, so a plain `split on |` false-positive cannot occur here. For each entry, emit a **Medium severity** finding: `naive-coherence — table row at line {entry.line} has {entry.actual_cols} columns; header has {entry.expected_cols}` (the `entry.section` and `entry.row` fields populate the detail when present).
|
|
87
59
|
|
|
88
60
|
**2.7 Scripts & Assets section.** If `{skillDir}/scripts/` or `{skillDir}/assets/` exists, `grep -n '^## Scripts' SKILL.md`:
|
|
89
61
|
- Directory exists AND no `## Scripts` section → **Medium severity** finding: `naive-coherence — scripts/assets directory exists but Scripts & Assets section missing` (per `{scoringRulesFile}`)
|
|
@@ -125,11 +97,11 @@ Scan SKILL.md for all cross-references:
|
|
|
125
97
|
- Integration pattern references (middleware chains, plugin hooks, shared state)
|
|
126
98
|
- Script/asset references (`scripts/{file}`, `assets/{file}`) in SKILL.md body
|
|
127
99
|
|
|
128
|
-
|
|
100
|
+
Delegate to a subagent that grep/regexes SKILL.md for reference patterns and returns ONLY this JSON shape — no prose, no commentary, no markdown fences: `{"references_found": [{"line": N, "type": "file-path|skill|type-import|integration-pattern|script-asset", "target": "..."}]}`. Parent strips wrapping markdown fences (if present) before parsing. If subagent unavailable, scan in main thread.
|
|
129
101
|
|
|
130
102
|
### 4. Contextual Mode: Validate Each Reference
|
|
131
103
|
|
|
132
|
-
|
|
104
|
+
For EACH reference found, delegate to a subagent that:
|
|
133
105
|
|
|
134
106
|
1. Checks if the target exists (file exists, skill exists, type is declared)
|
|
135
107
|
2. If target exists, validates the reference is accurate:
|
|
@@ -138,13 +110,13 @@ DO NOT BE LAZY — For EACH reference found, launch a subprocess that:
|
|
|
138
110
|
- Skill references: referenced skill exists in skills output folder
|
|
139
111
|
- Integration patterns: documented pattern matches actual implementation
|
|
140
112
|
- Script/asset references: verify the referenced file exists in the skill's `scripts/` or `assets/` directory
|
|
141
|
-
3. Returns
|
|
113
|
+
3. Returns ONLY this JSON shape per reference — no prose, no commentary, no markdown fences: `{"reference": "...", "line": N, "target_exists": <bool>, "type_match": <bool>, "signature_match": <bool>, "issues": ["..."]}`
|
|
142
114
|
|
|
143
|
-
If
|
|
115
|
+
Parent strips wrapping markdown fences (if present) before parsing. If subagent unavailable, validate each reference in main thread.
|
|
144
116
|
|
|
145
117
|
4. **Scripts/assets directory check:** If a `scripts/` or `assets/` directory exists alongside SKILL.md, verify that a "Scripts & Assets" section (Section 7b) is present in SKILL.md. This directory-level check applies in both modes (naive mode performs it in Section 2; contextual mode performs it here alongside per-reference validation). Flag absence as Medium severity gap per `{scoringRulesFile}`.
|
|
146
118
|
|
|
147
|
-
5. **Path containment
|
|
119
|
+
5. **Path containment:** for every resolved reference target, compute its canonical path (`os.path.realpath`) and require that it lives inside `{skillDir}` OR inside `{source_path}` (the extraction tree recorded in metadata.json). References whose canonical path escapes both roots (e.g. `../../../etc/passwd`, absolute paths to unrelated dirs, symlink redirections outside the skill or its source) are **High severity** findings: `coherence — reference escapes skill/source sandbox: {raw_ref} → {canonical_path}`. Do NOT validate the target's contents for escaping references — the escape itself is the finding.
|
|
148
120
|
|
|
149
121
|
### 5. Contextual Mode: Check Integration Pattern Completeness
|
|
150
122
|
|
|
@@ -220,20 +192,5 @@ Reference validation of **{skill_name}**:
|
|
|
220
192
|
|
|
221
193
|
**Proceeding to external validation...**"
|
|
222
194
|
|
|
223
|
-
|
|
224
|
-
|
|
225
|
-
Display: "**Proceeding to external validation...**"
|
|
226
|
-
|
|
227
|
-
#### Menu Handling Logic:
|
|
228
|
-
|
|
229
|
-
- After coherence analysis is complete, update {outputFile} frontmatter stepsCompleted, then immediately load, read entire file, then execute {nextStepFile}
|
|
230
|
-
|
|
231
|
-
#### EXECUTION RULES:
|
|
232
|
-
|
|
233
|
-
- This is an auto-proceed validation step with no user choices
|
|
234
|
-
- Proceed directly to next step after coherence is analyzed
|
|
235
|
-
|
|
236
|
-
## CRITICAL STEP COMPLETION NOTE
|
|
237
|
-
|
|
238
|
-
ONLY WHEN coherence analysis is complete (naive structural or contextual full validation), the Coherence Analysis section has been appended to {outputFile}, and coherence scores (if contextual) have been calculated, will you then load and read fully `{nextStepFile}` to execute external validation.
|
|
195
|
+
Update stepsCompleted, then load and execute {nextStepFile}.
|
|
239
196
|
|
package/src/skf-test-skill/{steps-c/step-03-coverage-check.md → references/coverage-check.md}
RENAMED
|
@@ -1,31 +1,23 @@
|
|
|
1
1
|
---
|
|
2
|
-
nextStepFile: '
|
|
2
|
+
nextStepFile: 'coherence-check.md'
|
|
3
3
|
outputFile: '{forge_version}/test-report-{skill_name}-{run_id}.md'
|
|
4
4
|
scoringRulesFile: 'references/scoring-rules.md'
|
|
5
5
|
sourceAccessProtocol: 'references/source-access-protocol.md'
|
|
6
6
|
---
|
|
7
7
|
|
|
8
|
+
<!-- Config: communicate in {communication_language}. -->
|
|
9
|
+
|
|
8
10
|
# Step 3: Coverage Check
|
|
9
11
|
|
|
10
12
|
## STEP GOAL:
|
|
11
13
|
|
|
12
14
|
Compare the exports, functions, classes, types, and interfaces documented in SKILL.md against the actual source code API surface. Identify missing documentation, undocumented exports, and signature mismatches. Analysis depth scales with forge tier.
|
|
13
15
|
|
|
14
|
-
## Rules
|
|
15
|
-
|
|
16
|
-
- Use subprocess optimization for per-file AST analysis when available; if unavailable, analyze sequentially
|
|
17
|
-
- For each source file, launch a subprocess for deep analysis — do not shortcut
|
|
18
|
-
- Coverage depth must match the detected forge tier
|
|
19
|
-
|
|
20
|
-
## MANDATORY SEQUENCE
|
|
21
|
-
|
|
22
|
-
**CRITICAL:** Follow this sequence exactly. Do not skip, reorder, or improvise unless user explicitly requests a change.
|
|
23
|
-
|
|
24
16
|
### 0. Check for Docs-Only Mode
|
|
25
17
|
|
|
26
18
|
**If all SKILL.md citations are `[EXT:...]` format (no local source citations):**
|
|
27
19
|
|
|
28
|
-
Set `docs_only_mode: true` in context for step
|
|
20
|
+
Set `docs_only_mode: true` in context for step 5 scoring. Coverage scoring adapts: instead of comparing SKILL.md against source code exports, compare SKILL.md documented items against themselves for internal completeness (every documented function has a description, parameters, and return type). Score based on documentation completeness rather than source coverage.
|
|
29
21
|
|
|
30
22
|
**Quick-tier weight adjustment:** If `confidence_tier` is also `"Quick"`, apply Quick-tier weight redistribution (zeroing Signature Accuracy and Type Coverage) as an additional step per `{scoringRulesFile}`.
|
|
31
23
|
|
|
@@ -80,13 +72,13 @@ test-skill is a quality gate — it MUST NOT trust subagent output blindly. Befo
|
|
|
80
72
|
**Schema validation (required keys + types):**
|
|
81
73
|
|
|
82
74
|
1. Strip wrapping markdown fences before parsing. Subagents frequently return JSON wrapped in a code fence — a line of three backticks (optionally followed by a language tag like `json`) preceding the JSON and a closing line of three backticks after it — despite prompt instructions to return raw JSON. When the first non-empty line of the response is three backticks (optionally with a language tag) and the last non-empty line is three backticks, remove those two fence lines before parsing. Then parse the remaining content as JSON. On parse failure of the inner content → HALT "coverage-check: subagent response not valid JSON".
|
|
83
|
-
2. Required keys present: `exports` (list), `cross_check_mismatches` (list — may be empty). Missing key or wrong type → HALT "coverage-check: subagent JSON schema invalid — missing/typo: {key}". Note: the parent already knows the skill name from workflow context (`{resolved_skill_package}` from step
|
|
75
|
+
2. Required keys present: `exports` (list), `cross_check_mismatches` (list — may be empty). Missing key or wrong type → HALT "coverage-check: subagent JSON schema invalid — missing/typo: {key}". Note: the parent already knows the skill name from workflow context (`{resolved_skill_package}` from step 1) — the subagent is not required to echo it back, and doing so introduces a contract-drift surface without improving verification.
|
|
84
76
|
3. Each `exports[]` entry must be a dict with at minimum `name` (non-empty string) and `kind` (one of `function|class|type|constant|hook|interface|method`). Reject entries violating this; if >0 rejections, HALT "coverage-check: subagent returned malformed export entries — {count} entries do not match schema".
|
|
85
77
|
4. `cross_check_mismatches[]` entries (when non-empty) must carry `export`, `skill_md_line`, `reference_file`, `reference_line`, `issue`. Missing fields → HALT.
|
|
86
78
|
|
|
87
79
|
**Spot-check (ground-truth verification, zero-hallucination guard):**
|
|
88
80
|
|
|
89
|
-
1. If `len(exports) == 0`: skip the spot-check (no names to verify). Zero-exports policy is handled in
|
|
81
|
+
1. If `len(exports) == 0`: skip the spot-check (no names to verify). Zero-exports policy is handled in the §2b zero-exports guard.
|
|
90
82
|
2. Otherwise, sample `min(3, len(exports))` exports deterministically — by default take indices `[0, len//2, len-1]` (first, middle, last) from the `exports` array after a stable sort by `name`.
|
|
91
83
|
3. For each sampled export, run: `grep -n "{export.name}" {resolved_skill_package}/SKILL.md` in the parent context. The name MUST appear at least once.
|
|
92
84
|
4. If ANY sampled name returns zero matches, HALT "coverage-check: subagent inventory failed ground-truth spot-check — `{name}` claimed as export but absent from SKILL.md".
|
|
@@ -125,7 +117,7 @@ Parent reads `cross_check_mismatches` from the subagent JSON summary. Build the
|
|
|
125
117
|
}
|
|
126
118
|
```
|
|
127
119
|
|
|
128
|
-
Flag each mismatch as **High severity** — signature inconsistency between SKILL.md body and reference files undermines agent trust. These findings feed into the gap report (step
|
|
120
|
+
Flag each mismatch as **High severity** — signature inconsistency between SKILL.md body and reference files undermines agent trust. These findings feed into the gap report (step 6).
|
|
129
121
|
|
|
130
122
|
### 2. Analyze Source Code (Tier-Dependent)
|
|
131
123
|
|
|
@@ -138,10 +130,10 @@ Start from the package entry point (see 0b) and identify the public API surface.
|
|
|
138
130
|
- Cannot verify signatures — note as "unverified" in report
|
|
139
131
|
|
|
140
132
|
**Forge Tier (ast-grep available):**
|
|
141
|
-
|
|
133
|
+
For EACH source file that defines public API exports, delegate to a subagent that:
|
|
142
134
|
1. Uses ast-grep to extract all exported symbols with their full signatures
|
|
143
135
|
2. Matches each export against the documented inventory
|
|
144
|
-
3. Returns
|
|
136
|
+
3. Returns ONLY the JSON object below — no prose, no commentary, no markdown fences:
|
|
145
137
|
|
|
146
138
|
```json
|
|
147
139
|
{
|
|
@@ -160,7 +152,7 @@ DO NOT BE LAZY — For EACH source file that defines public API exports, launch
|
|
|
160
152
|
}
|
|
161
153
|
```
|
|
162
154
|
|
|
163
|
-
If
|
|
155
|
+
Parent strips wrapping markdown fences (if present) before parsing, same as §1a. If subagent unavailable, perform ast-grep analysis in main thread per file.
|
|
164
156
|
|
|
165
157
|
**Deep Tier (ast-grep + gh + QMD):**
|
|
166
158
|
- All Forge tier checks, plus:
|
|
@@ -168,7 +160,7 @@ If subprocess unavailable, perform ast-grep analysis in main thread per file.
|
|
|
168
160
|
- Cross-check type definitions against their source declarations
|
|
169
161
|
- Verify re-exported symbols trace to their original source
|
|
170
162
|
|
|
171
|
-
### 2b. Zero-Exports Guard
|
|
163
|
+
### 2b. Zero-Exports Guard
|
|
172
164
|
|
|
173
165
|
After the source-code analysis (§2) completes, compute `total_exports` — the count of exports discovered in the source / provenance-map / metadata.json, per the stratified-scope and State 2 rules resolved in §4.
|
|
174
166
|
|
|
@@ -225,7 +217,7 @@ Load `{scoringRulesFile}` to determine category scores:
|
|
|
225
217
|
|
|
226
218
|
Record the denominator source in the Coverage Analysis section as `Denominator: stratified ({effective_denominator | tier_a_include union | scope.include union}, {N} files matched)`. When stratified scope does not apply, use the standard barrel-based denominator and omit the stratified annotation.
|
|
227
219
|
|
|
228
|
-
**
|
|
220
|
+
**Record the two non-chosen candidate values alongside the chosen one.**
|
|
229
221
|
Stratified-scope resolution picks ONE of three denominator candidates
|
|
230
222
|
(`stats.effective_denominator`, `tier_a_include` union, `scope.include` union)
|
|
231
223
|
per the priority above. To make the choice auditable, append a
|
|
@@ -234,7 +226,7 @@ all three values — the chosen one explicitly marked and the other two recorded
|
|
|
234
226
|
as-observed (or `absent` when the candidate was not present for this skill):
|
|
235
227
|
|
|
236
228
|
```markdown
|
|
237
|
-
**Denominator Candidates** (
|
|
229
|
+
**Denominator Candidates** (stratified-scope audit trail):
|
|
238
230
|
- `stats.effective_denominator`: {N | absent} {← chosen if priority (1) applied}
|
|
239
231
|
- `scope.tier_a_include` union: {N | absent} {← chosen if priority (2) applied}
|
|
240
232
|
- `scope.include` union: {N | absent} {← chosen if priority (3) applied}
|
|
@@ -262,7 +254,7 @@ After the denominator has been resolved (standard, stratified, or State 2), cros
|
|
|
262
254
|
3. `metadata.json.stats.exports_documented` — the declared documented count
|
|
263
255
|
4. Provenance-map entry count (if `{forge_data_folder}/{skill_name}/provenance-map.json` exists)
|
|
264
256
|
|
|
265
|
-
Cluster assignment is canonical: `skf-create-skill` step
|
|
257
|
+
Cluster assignment is canonical: `skf-create-skill` step 5 derives `exports_public_api` from entry-point validation and writes the `exports[]` array from the same barrel surface (see `skf-create-skill/references/compile.md:105`), while `exports_documented` tracks the broader documented surface that the provenance-map also enumerates.
|
|
266
258
|
|
|
267
259
|
**Intra-cluster divergence (Medium):** For each cluster, if two counts are present and disagree by more than 10% of the larger, emit a **Medium**-severity gap titled `metadata drift — {cluster} export counts diverge` (substitute `barrel` for Cluster A, `documented-surface` for Cluster B). Enumerate the offending counts in the gap body (e.g., `stats.exports_public_api=55, exports[].length=48` → 13% drift). This is the real drift signal — the two sources should mirror the same surface and they don't, so upstream extraction or compilation produced inconsistent output that a re-compile should reconcile. Classify under structural/metadata coherence regardless of naive/contextual mode.
|
|
268
260
|
|
|
@@ -311,7 +303,7 @@ Append the **Coverage Analysis** section to `{outputFile}`:
|
|
|
311
303
|
| Signature Accuracy | {N}% or N/A |
|
|
312
304
|
| Type Coverage | {N}% or N/A |
|
|
313
305
|
|
|
314
|
-
Note: Weight application is deferred to step
|
|
306
|
+
Note: Weight application is deferred to step 5 where all category weights are calculated after external validation availability is known.
|
|
315
307
|
```
|
|
316
308
|
|
|
317
309
|
### 6. Report Coverage Results
|
|
@@ -328,20 +320,5 @@ Note: Weight application is deferred to step-05 where all category weights are c
|
|
|
328
320
|
|
|
329
321
|
**Proceeding to coherence check...**"
|
|
330
322
|
|
|
331
|
-
|
|
332
|
-
|
|
333
|
-
Display: "**Proceeding to coherence check...**"
|
|
334
|
-
|
|
335
|
-
#### Menu Handling Logic:
|
|
336
|
-
|
|
337
|
-
- After coverage analysis is complete, update {outputFile} frontmatter stepsCompleted, then immediately load, read entire file, then execute {nextStepFile}
|
|
338
|
-
|
|
339
|
-
#### EXECUTION RULES:
|
|
340
|
-
|
|
341
|
-
- This is an auto-proceed validation step with no user choices
|
|
342
|
-
- Proceed directly to next step after coverage is analyzed
|
|
343
|
-
|
|
344
|
-
## CRITICAL STEP COMPLETION NOTE
|
|
345
|
-
|
|
346
|
-
ONLY WHEN all source files have been analyzed, the Coverage Analysis section has been appended to {outputFile}, and category scores have been calculated, will you then load and read fully `{nextStepFile}` to execute coherence check.
|
|
323
|
+
Update stepsCompleted, then load and execute {nextStepFile}.
|
|
347
324
|
|
|
@@ -1,23 +1,16 @@
|
|
|
1
1
|
---
|
|
2
|
-
nextStepFile: '
|
|
2
|
+
nextStepFile: 'coverage-check.md'
|
|
3
3
|
outputFile: '{forge_version}/test-report-{skill_name}-{run_id}.md'
|
|
4
4
|
---
|
|
5
5
|
|
|
6
|
+
<!-- Config: communicate in {communication_language}. -->
|
|
7
|
+
|
|
6
8
|
# Step 2: Detect Mode
|
|
7
9
|
|
|
8
10
|
## STEP GOAL:
|
|
9
11
|
|
|
10
12
|
Examine the skill metadata to determine whether this is an individual skill (naive mode — API surface coverage only) or a stack skill (contextual mode — full coherence validation including cross-references and integration patterns).
|
|
11
13
|
|
|
12
|
-
## Rules
|
|
13
|
-
|
|
14
|
-
- Focus only on mode detection — do not begin coverage or coherence analysis
|
|
15
|
-
- Set testMode in output frontmatter before proceeding
|
|
16
|
-
|
|
17
|
-
## MANDATORY SEQUENCE
|
|
18
|
-
|
|
19
|
-
**CRITICAL:** Follow this sequence exactly. Do not skip, reorder, or improvise unless user explicitly requests a change.
|
|
20
|
-
|
|
21
14
|
### 1. Examine Skill Type
|
|
22
15
|
|
|
23
16
|
Read the skill metadata (loaded in step 01) and check:
|
|
@@ -86,20 +79,5 @@ Append the **Test Summary** section to `{outputFile}`:
|
|
|
86
79
|
|
|
87
80
|
**Proceeding to coverage check...**"
|
|
88
81
|
|
|
89
|
-
|
|
90
|
-
|
|
91
|
-
Display: "**Proceeding to coverage check...**"
|
|
92
|
-
|
|
93
|
-
#### Menu Handling Logic:
|
|
94
|
-
|
|
95
|
-
- After mode detection is complete, update {outputFile} frontmatter stepsCompleted, then immediately load, read entire file, then execute {nextStepFile}
|
|
96
|
-
|
|
97
|
-
#### EXECUTION RULES:
|
|
98
|
-
|
|
99
|
-
- This is an auto-proceed detection step with no user choices
|
|
100
|
-
- Proceed directly to next step after mode is set
|
|
101
|
-
|
|
102
|
-
## CRITICAL STEP COMPLETION NOTE
|
|
103
|
-
|
|
104
|
-
ONLY WHEN testMode is set in the output frontmatter and the Test Summary section is appended to the output document, will you then load and read fully `{nextStepFile}` to execute coverage check.
|
|
82
|
+
Update stepsCompleted, then load and execute {nextStepFile}.
|
|
105
83
|
|
|
@@ -1,24 +1,16 @@
|
|
|
1
1
|
---
|
|
2
|
-
nextStepFile: '
|
|
2
|
+
nextStepFile: 'score.md'
|
|
3
3
|
outputFile: '{forge_version}/test-report-{skill_name}-{run_id}.md'
|
|
4
4
|
---
|
|
5
5
|
|
|
6
|
+
<!-- Config: communicate in {communication_language}. -->
|
|
7
|
+
|
|
6
8
|
# Step 4b: External Validators
|
|
7
9
|
|
|
8
10
|
## STEP GOAL:
|
|
9
11
|
|
|
10
12
|
Run external validation tools (`skill-check` and `tessl`) against the skill directory, capture their scores and findings, and append results to the test report. These tools catch complementary issues that internal coverage and coherence checks miss: `skill-check` validates spec compliance while `tessl` evaluates content quality and actionability.
|
|
11
13
|
|
|
12
|
-
## Rules
|
|
13
|
-
|
|
14
|
-
- Run each tool independently — one tool's failure does not affect the other
|
|
15
|
-
- Do not halt the workflow if either tool is unavailable or fails
|
|
16
|
-
- Read-only — do not modify skill content
|
|
17
|
-
|
|
18
|
-
## MANDATORY SEQUENCE
|
|
19
|
-
|
|
20
|
-
**CRITICAL:** Follow this sequence exactly. Do not skip, reorder, or improvise.
|
|
21
|
-
|
|
22
14
|
### 1. Resolve Skill Directory
|
|
23
15
|
|
|
24
16
|
Read {outputFile} frontmatter to get the skill directory path (`skillDir`).
|
|
@@ -65,7 +57,7 @@ on the actual validator run below. If the probe exits non-zero OR the 15s
|
|
|
65
57
|
timeout trips (exit code `124`), record `skill_check_score: N/A` and skip to
|
|
66
58
|
section 3.
|
|
67
59
|
|
|
68
|
-
**Run validation (
|
|
60
|
+
**Run validation (120s timeout):**
|
|
69
61
|
|
|
70
62
|
```bash
|
|
71
63
|
timeout 120s npx skill-check check {skillDir} --format json --no-security-scan
|
|
@@ -97,7 +89,7 @@ prevent a cold-cache fetch from stalling the workflow. If the probe exits
|
|
|
97
89
|
non-zero OR the 15s timeout trips (exit code `124`), record
|
|
98
90
|
`tessl_score: N/A` and skip to section 4.
|
|
99
91
|
|
|
100
|
-
**Run review (
|
|
92
|
+
**Run review (120s timeout):**
|
|
101
93
|
|
|
102
94
|
The §2 probe (`npx --no-install -y tessl --version`) already resolved tessl via the caller's npm cache or a locally-installed binary on `$PATH`. Invoke the same binary for the review — do not re-pin to a registry-published version.
|
|
103
95
|
|
|
@@ -185,19 +177,5 @@ Append to `{outputFile}`:
|
|
|
185
177
|
|
|
186
178
|
**Proceeding to scoring...**"
|
|
187
179
|
|
|
188
|
-
|
|
189
|
-
|
|
190
|
-
#### Menu Handling Logic:
|
|
191
|
-
|
|
192
|
-
- After external validation is complete, update {outputFile} frontmatter stepsCompleted, then immediately load, read entire file, then execute {nextStepFile}
|
|
193
|
-
|
|
194
|
-
#### EXECUTION RULES:
|
|
195
|
-
|
|
196
|
-
- This is an auto-proceed validation step with no user choices
|
|
197
|
-
- Tool unavailability is a skip, not a halt
|
|
198
|
-
- Proceed directly to next step after external validation
|
|
199
|
-
|
|
200
|
-
## CRITICAL STEP COMPLETION NOTE
|
|
201
|
-
|
|
202
|
-
ONLY WHEN external validation results are appended to {outputFile} and scores are stored in context will you proceed to load `{nextStepFile}` for scoring.
|
|
180
|
+
Update stepsCompleted, then load and execute {nextStepFile}.
|
|
203
181
|
|
|
@@ -0,0 +1,14 @@
|
|
|
1
|
+
---
|
|
2
|
+
# `{nextStepFile}` is resolved by probing both candidate roots in order.
|
|
3
|
+
# HALT if neither exists — step 6 §7 should have caught this already, but
|
|
4
|
+
# this step re-asserts the invariant at dispatch time.
|
|
5
|
+
nextStepFileProbeOrder:
|
|
6
|
+
- '{project-root}/_bmad/skf/shared/health-check.md'
|
|
7
|
+
- '{project-root}/src/shared/health-check.md'
|
|
8
|
+
---
|
|
9
|
+
|
|
10
|
+
<!-- Config: communicate in {communication_language}. -->
|
|
11
|
+
|
|
12
|
+
# Step 7: Workflow Health Check
|
|
13
|
+
|
|
14
|
+
Probe `{nextStepFileProbeOrder}` in order; load and execute the first path that exists as `{nextStepFile}`, else HALT with a diagnostic naming both candidate paths. This is the terminal step of test-skill.
|
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
---
|
|
2
|
-
nextStepFile: '
|
|
2
|
+
nextStepFile: 'detect-mode.md'
|
|
3
3
|
outputFile: '{forge_version}/test-report-{skill_name}-{run_id}.md'
|
|
4
4
|
templateFile: 'templates/test-report-template.md'
|
|
5
5
|
sidecarFile: '{sidecar_path}/forge-tier.yaml'
|
|
@@ -13,27 +13,24 @@ frontmatterScriptProbeOrder:
|
|
|
13
13
|
versionPathsKnowledge: 'knowledge/version-paths.md'
|
|
14
14
|
---
|
|
15
15
|
|
|
16
|
+
<!-- Config: communicate in {communication_language}. -->
|
|
17
|
+
|
|
16
18
|
# Step 1: Initialize Test
|
|
17
19
|
|
|
18
20
|
## STEP GOAL:
|
|
19
21
|
|
|
20
22
|
Discover and validate the target skill, load forge tier state to determine analysis depth, and create the test report document from template.
|
|
21
23
|
|
|
22
|
-
## Rules
|
|
23
|
-
|
|
24
|
-
- Focus only on input discovery and validation — do not begin analysis
|
|
25
|
-
- Validate all required files exist before proceeding
|
|
26
|
-
|
|
27
|
-
## MANDATORY SEQUENCE
|
|
28
|
-
|
|
29
|
-
**CRITICAL:** Follow this sequence exactly. Do not skip, reorder, or improvise unless user explicitly requests a change.
|
|
30
|
-
|
|
31
24
|
### 1. Receive Skill Path
|
|
32
25
|
|
|
33
26
|
If skill path was provided as workflow argument, use it directly.
|
|
34
27
|
|
|
35
28
|
**Recognized flags on the invocation:**
|
|
36
29
|
- `--allow-workspace-drift` — bypass the section 5b pre-flight guard that halts when local workspace HEAD does not match `metadata.source_commit`. Store `allow_workspace_drift: true` in workflow context when present. No effect when `source_commit` is unpinned or the source is not a git working tree.
|
|
30
|
+
- `--no-discovery` — skip the §4b Discovery Testing block in step 6 (report). Store `no_discovery: true` in workflow context when present.
|
|
31
|
+
- `--no-health-check` — skip the §7 health-check dispatch in step 6 (report). Store `no_health_check: true` in workflow context when present.
|
|
32
|
+
- `--tier=<Quick|Forge|Forge+|Deep>` — bypass the §4 forge-tier.yaml sidecar HALT. Store `tier_flag: '<value>'` in workflow context when present; §4 will set `detected_tier` directly from this value and skip the sidecar probe.
|
|
33
|
+
- `--threshold=<N>` — override the pass threshold for this run. Consumed by `references/score.md` §1; CLI wins over the `workflow.default_threshold` scalar.
|
|
37
34
|
|
|
38
35
|
If no path provided, ask:
|
|
39
36
|
|
|
@@ -99,10 +96,10 @@ timeout 30s uv run {frontmatterScript} {resolved_skill_package}/SKILL.md --skill
|
|
|
99
96
|
|
|
100
97
|
If the command trips the 30s wall-clock (exit code `124`), set
|
|
101
98
|
`analysis_confidence: degraded` and `toolingStatus: frontmatter-validator-timeout`
|
|
102
|
-
in workflow context, apply the step
|
|
99
|
+
in workflow context, apply the step 5 tooling-degraded cap (score capped at
|
|
103
100
|
`threshold - 1` → auto-FAIL), and record the reason in evidence-report.
|
|
104
101
|
|
|
105
|
-
Parse the JSON output.
|
|
102
|
+
Parse the JSON output. Treat each `status` value explicitly:
|
|
106
103
|
|
|
107
104
|
- `status: "pass"` — continue silently.
|
|
108
105
|
- `status: "warn"` — display the warning below, log each issue as a pre-check finding, and continue with testing. Frontmatter issues surface in the gap report alongside coverage/coherence findings.
|
|
@@ -116,18 +113,20 @@ Parse the JSON output. Per B2, treat each `status` value explicitly:
|
|
|
116
113
|
This skill will fail `npx skills add` and `npx skill-check check`. {If warn:} Consider fixing frontmatter before proceeding (run `npx skill-check check <skill-dir> --fix` to auto-fix deterministic issues). {If fail:} test-skill cannot proceed — halt and repair frontmatter, then re-run.
|
|
117
114
|
```
|
|
118
115
|
|
|
119
|
-
**3c. Python runtime probe.** Before the first invocation, confirm both `python3` AND `uv` are on `$PATH` (`command -v python3` and `command -v uv`). Both are required: `uv run` shells through to `python3` under the hood AND honors the script's PEP 723 PyYAML dependency declaration that bare `python3` ignores. If either is missing, set `analysis_confidence: degraded` in workflow context and carry a **score cap** into step
|
|
116
|
+
**3c. Python runtime probe.** Before the first invocation, confirm both `python3` AND `uv` are on `$PATH` (`command -v python3` and `command -v uv`). Both are required: `uv run` shells through to `python3` under the hood AND honors the script's PEP 723 PyYAML dependency declaration that bare `python3` ignores. If either is missing, set `analysis_confidence: degraded` in workflow context and carry a **score cap** into step 5: `capped_score = threshold - 1` → forces auto-FAIL until the runtime is restored. Record the reason in evidence-report and the test report frontmatter (`analysisConfidence: degraded`, `toolingStatus: python3-missing` or `uv-missing` as appropriate). `uv` is a documented runtime prerequisite — see `docs/getting-started.md` for install instructions.
|
|
120
117
|
|
|
121
118
|
### 4. Load Forge Tier State
|
|
122
119
|
|
|
123
|
-
|
|
120
|
+
**`--tier=<...>` flag bypass (precedes the sidecar probe).** If `tier_flag` is set in workflow context (from §1's `--tier=<Quick|Forge|Forge+|Deep>` flag), validate the value against the allowed set. On valid match: set `detected_tier` directly to the flag's value, leave `ast_grep`/`gh_cli`/`qmd` availability flags unset (downstream steps treat unset as "unknown" — analysis proceeds without tool-specific enrichment), log Info note "tier — supplied via --tier flag, sidecar bypassed", and SKIP the sidecar probe and HALT below (jump straight to §4b "Apply Tier Override"). On invalid value (not one of the four), HALT with "Error: --tier=<value> is not one of Quick, Forge, Forge+, Deep".
|
|
121
|
+
|
|
122
|
+
**Otherwise (no `--tier` flag):** Read `{sidecarFile}` to determine available analysis depth.
|
|
124
123
|
|
|
125
124
|
**If forge-tier.yaml exists:**
|
|
126
125
|
- Read `tier` value (Quick, Forge, Forge+, or Deep)
|
|
127
126
|
- Read tool availability flags (ast_grep, gh_cli, qmd)
|
|
128
127
|
|
|
129
128
|
**If forge-tier.yaml missing:**
|
|
130
|
-
"**Cannot proceed.** forge-tier.yaml not found at `{sidecarFile}`. Please run the **setup** workflow first to configure your forge tier (Quick/Forge/Forge+/Deep)."
|
|
129
|
+
"**Cannot proceed.** forge-tier.yaml not found at `{sidecarFile}`. Please run the **setup** workflow first to configure your forge tier (Quick/Forge/Forge+/Deep), or re-run with `--tier=<Quick|Forge|Forge+|Deep>` to bypass the sidecar."
|
|
131
130
|
|
|
132
131
|
HALT — do not proceed.
|
|
133
132
|
|
|
@@ -154,7 +153,7 @@ Test-skill reads `source_path` during coverage and coherence analysis. If the lo
|
|
|
154
153
|
|
|
155
154
|
- Resolve `pinned_commit` from `metadata.source_commit`.
|
|
156
155
|
- **If `pinned_commit` is null, empty, or `"local"`:** skip the guard; log `workspace_drift_check: skipped (no pinned commit)` and continue to section 6.
|
|
157
|
-
- **If `pinned_commit` is a per-repo map (stack skills):** iterate each `{repo_path: commit}` entry — for each repo run `git -C "{repo_path}" rev-parse HEAD` and compare to its pinned commit (accept full-SHA or short-SHA-prefix match). If ANY repo diverges and the user did not pass `--allow-workspace-drift`, HALT with exit status `halted-for-workspace-drift` listing every mismatched repo. On all-match: log `workspace_drift_check: ok (stack, {N} repos verified)` and continue to section 6.
|
|
156
|
+
- **If `pinned_commit` is a per-repo map (stack skills):** iterate each `{repo_path: commit}` entry — for each repo run `git -C "{repo_path}" rev-parse HEAD` and compare to its pinned commit (accept full-SHA or short-SHA-prefix match). If ANY repo diverges and the user did not pass `--allow-workspace-drift`, HALT with exit status `halted-for-workspace-drift` listing every mismatched repo. On all-match: log `workspace_drift_check: ok (stack, {N} repos verified)` and continue to section 6. This guard MUST iterate every repo — do not skip stack skills.
|
|
158
157
|
- **If `source_path` is not a git working tree** (bare checkout, tarball extract, docs-only source) — detect by `git -C "{source_path}" rev-parse --is-inside-work-tree`, non-zero exit means skip: log `workspace_drift_check: skipped (not a git working tree)` and continue to section 6.
|
|
159
158
|
- **Otherwise** run `git -C "{source_path}" rev-parse HEAD` and compare to `pinned_commit`. Accept full-SHA or short-SHA-prefix match (stored pins are often 8-char short hashes — see `src/knowledge/provenance-tracking.md`).
|
|
160
159
|
- **On match:** log `workspace_drift_check: ok ({short_sha})` and continue.
|
|
@@ -177,15 +176,15 @@ Test-skill reads `source_path` during coverage and coherence analysis. If the lo
|
|
|
177
176
|
```
|
|
178
177
|
|
|
179
178
|
Do not proceed. The test report has not been created; no partial writes.
|
|
180
|
-
- **On mismatch WITH `--allow-workspace-drift`:** log `workspace_drift_check: overridden (pinned={pinned_commit}, head={head_sha})`, carry the warning into the final report frontmatter (`workspaceDrift: overridden`), and set `allow_workspace_drift: true` in workflow context (consumed by step
|
|
179
|
+
- **On mismatch WITH `--allow-workspace-drift`:** log `workspace_drift_check: overridden (pinned={pinned_commit}, head={head_sha})`, carry the warning into the final report frontmatter (`workspaceDrift: overridden`), and set `allow_workspace_drift: true` in workflow context (consumed by step 5 §5 drift override — a PASS under drift is demoted to `pass-with-drift` and `nextWorkflow` is forced to `update-skill`, never `export-skill`). Continue.
|
|
181
180
|
|
|
182
181
|
### 6. Create Output Document
|
|
183
182
|
|
|
184
|
-
**6a. Generate `{run_id}
|
|
183
|
+
**6a. Generate `{run_id}`**: a per-run identifier of the form `{YYYYMMDDTHHmmssZ}-{pid}-{rand4}` (UTC timestamp + process PID + 4-char random hex). Store in workflow context. All per-run artifacts in this and subsequent steps MUST carry this suffix; step 6 verifies `testDate` in the resulting report matches the run's stamp and fail-fast otherwise.
|
|
185
184
|
|
|
186
|
-
**6b. Acquire the per-skill test lock
|
|
185
|
+
**6b. Acquire the per-skill test lock**: `flock {forge_version}/.test-skill.lock` for the duration of this run to serialize concurrent `skf-test-skill` invocations against the same skill. If the lock is already held by another run, HALT with "another test-skill run is active for {skill_name}".
|
|
187
186
|
|
|
188
|
-
**6c. Create `{outputFile}` from `{templateFile}`** — use `{forge_version}/test-report-{skill_name}-{run_id}.md`
|
|
187
|
+
**6c. Create `{outputFile}` from `{templateFile}`** — use `{forge_version}/test-report-{skill_name}-{run_id}.md` Initial frontmatter:
|
|
189
188
|
|
|
190
189
|
```yaml
|
|
191
190
|
---
|
|
@@ -202,7 +201,7 @@ analysisConfidence: '{full|degraded}'
|
|
|
202
201
|
toolingStatus: '{ok|python3-missing|uv-missing|frontmatter-validator-missing|frontmatter-validator-timeout}'
|
|
203
202
|
workspaceDrift: '{not-checked|ok|overridden}'
|
|
204
203
|
testDate: '{run_id timestamp ISO-8601 UTC}'
|
|
205
|
-
stepsCompleted: ['
|
|
204
|
+
stepsCompleted: ['init']
|
|
206
205
|
nextWorkflow: ''
|
|
207
206
|
---
|
|
208
207
|
```
|
|
@@ -219,20 +218,5 @@ nextWorkflow: ''
|
|
|
219
218
|
|
|
220
219
|
**Proceeding to mode detection...**"
|
|
221
220
|
|
|
222
|
-
|
|
223
|
-
|
|
224
|
-
Display: "**Proceeding to mode detection...**"
|
|
225
|
-
|
|
226
|
-
#### Menu Handling Logic:
|
|
227
|
-
|
|
228
|
-
- After initialization is complete, update {outputFile} frontmatter stepsCompleted, then immediately load, read entire file, then execute {nextStepFile}
|
|
229
|
-
|
|
230
|
-
#### EXECUTION RULES:
|
|
231
|
-
|
|
232
|
-
- This is an auto-proceed initialization step with no user choices
|
|
233
|
-
- Proceed directly to next step after setup
|
|
234
|
-
|
|
235
|
-
## CRITICAL STEP COMPLETION NOTE
|
|
236
|
-
|
|
237
|
-
ONLY WHEN all required files are validated, forge tier is loaded, output document is created, and initialization status is reported, will you then load and read fully `{nextStepFile}` to execute mode detection.
|
|
221
|
+
Update stepsCompleted, then load and execute {nextStepFile}.
|
|
238
222
|
|
|
@@ -1,6 +1,8 @@
|
|
|
1
|
+
<!-- Config: communicate in {communication_language}. -->
|
|
2
|
+
|
|
1
3
|
# Migration & Deprecation Section Rules (§2b / §5b)
|
|
2
4
|
|
|
3
|
-
> **Single source of truth.** Both `
|
|
5
|
+
> **Single source of truth.** Both `coherence-check.md` §2b (naive path)
|
|
4
6
|
> and §5b (contextual path) apply the rules in this file verbatim. Update this
|
|
5
7
|
> file — not the step file sections — when the rule set changes.
|
|
6
8
|
|
|
@@ -62,7 +64,7 @@ awk '/^---$/{c++;next} c==1 && /^t2_future_count:/{print $2; exit}' \
|
|
|
62
64
|
- **`t2_future_count` parsed** → use its integer value for the Case Rules
|
|
63
65
|
below.
|
|
64
66
|
|
|
65
|
-
The pinned field is emitted by `skf-create-skill/
|
|
67
|
+
The pinned field is emitted by `skf-create-skill/references/compile.md`
|
|
66
68
|
§7 (frontmatter-pinned fields), which ALWAYS writes `t2_future_count: N`
|
|
67
69
|
(including 0). Legacy skills whose `evidence-report.md` predates the pinned
|
|
68
70
|
field land in Case 4.
|