bmad-module-skill-forge 0.6.0 → 0.7.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +2 -2
- package/docs/agents.md +9 -5
- package/docs/concepts.md +17 -13
- package/docs/examples.md +19 -4
- package/docs/getting-started.md +34 -19
- package/docs/how-it-works.md +87 -84
- package/docs/index.md +1 -1
- package/docs/workflows.md +88 -40
- package/package.json +1 -1
- package/src/agents/forger.agent.yaml +17 -5
- package/src/forger/README.md +1 -1
- package/src/forger/forge-tier.yaml +2 -1
- package/src/forger/preferences.yaml +5 -7
- package/src/knowledge/agentskills-spec.md +1 -1
- package/src/knowledge/architecture-verification.md +102 -0
- package/src/knowledge/ccc-bridge.md +3 -2
- package/src/knowledge/confidence-tiers.md +12 -1
- package/src/knowledge/doc-fetcher.md +1 -1
- package/src/knowledge/manual-section-integrity.md +2 -2
- package/src/knowledge/overview.md +11 -7
- package/src/knowledge/progressive-capability.md +5 -2
- package/src/knowledge/qmd-registry.md +5 -2
- package/src/knowledge/skf-knowledge-index.csv +2 -0
- package/src/knowledge/skill-lifecycle.md +26 -2
- package/src/knowledge/split-body-strategy.md +16 -2
- package/src/knowledge/tool-resolution.md +55 -0
- package/src/knowledge/zero-hallucination.md +11 -8
- package/src/module-help.csv +9 -6
- package/src/module.yaml +3 -0
- package/src/workflows/README.md +35 -17
- package/src/workflows/analyze-source/data/skill-brief-schema.md +18 -5
- package/src/workflows/analyze-source/steps-c/step-01-init.md +2 -2
- package/src/workflows/analyze-source/steps-c/step-01b-continue.md +1 -1
- package/src/workflows/analyze-source/steps-c/step-02-scan-project.md +1 -1
- package/src/workflows/analyze-source/steps-c/step-03-identify-units.md +1 -1
- package/src/workflows/analyze-source/steps-c/step-04-map-and-detect.md +3 -3
- package/src/workflows/analyze-source/steps-c/step-05-recommend.md +1 -1
- package/src/workflows/analyze-source/steps-c/step-06-generate-briefs.md +2 -2
- package/src/workflows/analyze-source/workflow.md +2 -2
- package/src/workflows/audit-skill/data/severity-rules.md +1 -1
- package/src/workflows/audit-skill/steps-c/step-01-init.md +2 -1
- package/src/workflows/audit-skill/steps-c/step-02-re-index.md +8 -1
- package/src/workflows/audit-skill/steps-c/step-03-structural-diff.md +2 -2
- package/src/workflows/audit-skill/steps-c/step-04-semantic-diff.md +2 -2
- package/src/workflows/audit-skill/steps-c/step-05-severity-classify.md +1 -1
- package/src/workflows/audit-skill/workflow.md +2 -1
- package/src/workflows/brief-skill/data/scope-templates.md +10 -0
- package/src/workflows/brief-skill/data/skill-brief-schema.md +25 -23
- package/src/workflows/brief-skill/steps-c/step-01-gather-intent.md +1 -1
- package/src/workflows/brief-skill/steps-c/step-02-analyze-target.md +1 -1
- package/src/workflows/brief-skill/steps-c/step-05-write-brief.md +3 -0
- package/src/workflows/create-skill/data/extraction-patterns.md +7 -1
- package/src/workflows/create-skill/data/skill-sections.md +12 -11
- package/src/workflows/create-skill/data/source-resolution-protocols.md +2 -2
- package/src/workflows/create-skill/data/tier-degradation-rules.md +1 -1
- package/src/workflows/create-skill/steps-c/step-01-load-brief.md +14 -10
- package/src/workflows/create-skill/steps-c/step-02b-ccc-discover.md +4 -0
- package/src/workflows/create-skill/steps-c/step-03-extract.md +19 -19
- package/src/workflows/create-skill/steps-c/step-03b-fetch-temporal.md +3 -3
- package/src/workflows/create-skill/steps-c/step-03c-fetch-docs.md +3 -1
- package/src/workflows/create-skill/steps-c/step-04-enrich.md +3 -1
- package/src/workflows/create-skill/steps-c/step-05-compile.md +4 -3
- package/src/workflows/create-skill/steps-c/step-06-validate.md +2 -2
- package/src/workflows/create-skill/steps-c/step-07-generate-artifacts.md +33 -44
- package/src/workflows/create-skill/steps-c/step-08-report.md +25 -11
- package/src/workflows/create-skill/workflow.md +2 -2
- package/src/workflows/create-stack-skill/data/compose-mode-rules.md +65 -0
- package/src/workflows/create-stack-skill/data/manifest-patterns.md +23 -1
- package/src/workflows/create-stack-skill/data/stack-skill-template.md +28 -6
- package/src/workflows/create-stack-skill/steps-c/step-01-init.md +25 -7
- package/src/workflows/create-stack-skill/steps-c/step-02-detect-manifests.md +37 -3
- package/src/workflows/create-stack-skill/steps-c/step-03-rank-and-confirm.md +29 -3
- package/src/workflows/create-stack-skill/steps-c/step-04-parallel-extract.md +50 -7
- package/src/workflows/create-stack-skill/steps-c/step-05-detect-integrations.md +35 -3
- package/src/workflows/create-stack-skill/steps-c/step-06-compile-stack.md +5 -1
- package/src/workflows/create-stack-skill/steps-c/step-07-generate-output.md +53 -5
- package/src/workflows/create-stack-skill/steps-c/step-08-validate.md +3 -3
- package/src/workflows/create-stack-skill/steps-c/step-09-report.md +8 -2
- package/src/workflows/create-stack-skill/workflow.md +2 -2
- package/src/workflows/export-skill/data/managed-section-format.md +8 -1
- package/src/workflows/export-skill/data/snippet-format.md +1 -1
- package/src/workflows/export-skill/steps-c/step-04-update-context.md +7 -2
- package/src/workflows/export-skill/steps-c/step-05-token-report.md +1 -1
- package/src/workflows/export-skill/workflow.md +3 -1
- package/src/workflows/quick-skill/data/skill-template.md +18 -6
- package/src/workflows/quick-skill/steps-c/step-04-compile.md +26 -6
- package/src/workflows/quick-skill/steps-c/step-05-validate.md +1 -1
- package/src/workflows/quick-skill/steps-c/step-06-write.md +3 -0
- package/src/workflows/quick-skill/workflow.md +1 -1
- package/src/workflows/refine-architecture/data/refinement-rules.md +102 -0
- package/src/workflows/refine-architecture/steps-c/step-01-init.md +188 -0
- package/src/workflows/refine-architecture/steps-c/step-02-gap-analysis.md +179 -0
- package/src/workflows/refine-architecture/steps-c/step-03-issue-detection.md +182 -0
- package/src/workflows/refine-architecture/steps-c/step-04-improvements.md +169 -0
- package/src/workflows/refine-architecture/steps-c/step-05-compile.md +200 -0
- package/src/workflows/refine-architecture/steps-c/step-06-report.md +147 -0
- package/src/workflows/refine-architecture/workflow.md +61 -0
- package/src/workflows/setup-forge/steps-c/step-01-detect-and-tier.md +2 -1
- package/src/workflows/setup-forge/steps-c/step-01b-ccc-index.md +5 -3
- package/src/workflows/setup-forge/steps-c/step-02-write-config.md +15 -16
- package/src/workflows/setup-forge/steps-c/step-03-auto-index.md +1 -1
- package/src/workflows/setup-forge/steps-c/step-04-report.md +2 -2
- package/src/workflows/setup-forge/workflow.md +3 -2
- package/src/workflows/test-skill/data/scoring-rules.md +3 -0
- package/src/workflows/test-skill/data/source-access-protocol.md +1 -1
- package/src/workflows/test-skill/steps-c/step-01-init.md +1 -0
- package/src/workflows/test-skill/steps-c/step-02-detect-mode.md +2 -0
- package/src/workflows/test-skill/workflow.md +2 -2
- package/src/workflows/update-skill/steps-c/step-02-detect-changes.md +3 -3
- package/src/workflows/update-skill/steps-c/step-03-re-extract.md +8 -3
- package/src/workflows/update-skill/steps-c/step-05-validate.md +8 -12
- package/src/workflows/update-skill/steps-c/step-06-write.md +5 -2
- package/src/workflows/update-skill/workflow.md +1 -1
- package/src/workflows/verify-stack/data/coverage-patterns.md +45 -0
- package/src/workflows/verify-stack/data/feasibility-report-template.md +63 -0
- package/src/workflows/verify-stack/data/integration-verification-rules.md +73 -0
- package/src/workflows/verify-stack/steps-c/step-01-init.md +174 -0
- package/src/workflows/verify-stack/steps-c/step-02-coverage.md +181 -0
- package/src/workflows/verify-stack/steps-c/step-03-integrations.md +195 -0
- package/src/workflows/verify-stack/steps-c/step-04-requirements.md +168 -0
- package/src/workflows/verify-stack/steps-c/step-05-synthesize.md +191 -0
- package/src/workflows/verify-stack/steps-c/step-06-report.md +197 -0
- package/src/workflows/verify-stack/workflow.md +61 -0
- package/src/workflows/analyze-source/validation-report.md +0 -657
- package/src/workflows/analyze-source/workflow-plan-analyze-source.md +0 -385
- package/src/workflows/audit-skill/validation-report.md +0 -545
- package/src/workflows/audit-skill/workflow-plan-audit-skill.md +0 -318
- package/src/workflows/brief-skill/validation-report.md +0 -630
- package/src/workflows/brief-skill/workflow-plan-brief-skill.md +0 -360
- package/src/workflows/create-skill/validation-report.md +0 -189
- package/src/workflows/create-skill/workflow-plan-create-skill.md +0 -525
- package/src/workflows/create-stack-skill/validation-report.md +0 -401
- package/src/workflows/create-stack-skill/workflow-plan-create-stack-skill.md +0 -481
- package/src/workflows/export-skill/validation-report.md +0 -263
- package/src/workflows/export-skill/workflow-plan-export-skill.md +0 -369
- package/src/workflows/quick-skill/validation-report.md +0 -543
- package/src/workflows/quick-skill/workflow-plan-quick-skill.md +0 -405
- package/src/workflows/setup-forge/validation-report.md +0 -504
- package/src/workflows/setup-forge/workflow-plan-setup-forge.md +0 -430
- package/src/workflows/test-skill/validation-report.md +0 -390
- package/src/workflows/test-skill/workflow-plan-test-skill.md +0 -292
- package/src/workflows/update-skill/data/tier-degradation-rules.md +0 -46
- package/src/workflows/update-skill/validation-report.md +0 -559
- package/src/workflows/update-skill/workflow-plan-update-skill.md +0 -441
|
@@ -1,390 +0,0 @@
|
|
|
1
|
-
---
|
|
2
|
-
validationDate: 2026-02-26
|
|
3
|
-
completionDate: 2026-02-26
|
|
4
|
-
workflowName: test-skill
|
|
5
|
-
workflowPath: _bmad-output/bmb-creations/workflows/test-skill
|
|
6
|
-
validationStatus: COMPLETE
|
|
7
|
-
---
|
|
8
|
-
|
|
9
|
-
# Validation Report: test-skill
|
|
10
|
-
|
|
11
|
-
**Validation Started:** 2026-02-26
|
|
12
|
-
**Validation Completed:** 2026-02-26
|
|
13
|
-
**Validator:** BMAD Workflow Validation System
|
|
14
|
-
**Standards Version:** BMAD Workflow Standards
|
|
15
|
-
|
|
16
|
-
---
|
|
17
|
-
|
|
18
|
-
## File Structure & Size
|
|
19
|
-
|
|
20
|
-
### Folder Structure: PASS
|
|
21
|
-
|
|
22
|
-
```
|
|
23
|
-
test-skill/
|
|
24
|
-
├── workflow.md (57 lines)
|
|
25
|
-
├── data/
|
|
26
|
-
│ ├── scoring-rules.md (72 lines)
|
|
27
|
-
│ └── output-section-formats.md (97 lines)
|
|
28
|
-
├── steps-c/
|
|
29
|
-
│ ├── step-01-init.md (185 lines) ✅
|
|
30
|
-
│ ├── step-02-detect-mode.md (162 lines) ✅
|
|
31
|
-
│ ├── step-03-coverage-check.md (222 lines) ⚠️ Approaching limit
|
|
32
|
-
│ ├── step-04-coherence-check.md (249 lines) ⚠️ Approaching limit
|
|
33
|
-
│ ├── step-05-score.md (222 lines) ⚠️ Approaching limit
|
|
34
|
-
│ └── step-06-report.md (175 lines) ✅
|
|
35
|
-
└── templates/
|
|
36
|
-
└── test-report-template.md (24 lines)
|
|
37
|
-
```
|
|
38
|
-
|
|
39
|
-
### Required Files: PASS
|
|
40
|
-
|
|
41
|
-
- ✅ workflow.md exists
|
|
42
|
-
- ✅ All 6 design steps have corresponding files
|
|
43
|
-
- ✅ Sequential numbering (01-06) with no gaps
|
|
44
|
-
- ✅ Final step (step-06) exists
|
|
45
|
-
- ✅ Data folder with scoring rules and output formats
|
|
46
|
-
- ✅ Templates folder with output template
|
|
47
|
-
|
|
48
|
-
### File Size Analysis: PASS (after remediation)
|
|
49
|
-
|
|
50
|
-
| File | Lines | Status |
|
|
51
|
-
|------|-------|--------|
|
|
52
|
-
| step-01-init.md | 185 | ✅ Good |
|
|
53
|
-
| step-02-detect-mode.md | 162 | ✅ Good |
|
|
54
|
-
| step-03-coverage-check.md | 222 | ⚠️ Approaching limit |
|
|
55
|
-
| step-04-coherence-check.md | 249 | ⚠️ Approaching limit (fixed from 294) |
|
|
56
|
-
| step-05-score.md | 222 | ⚠️ Approaching limit |
|
|
57
|
-
| step-06-report.md | 175 | ✅ Good (fixed from 254) |
|
|
58
|
-
|
|
59
|
-
**Remediation applied:**
|
|
60
|
-
- Created `data/output-section-formats.md` — extracted output templates from step-04 and step-06
|
|
61
|
-
- step-04: 294 → 249 lines (output format templates moved to data file)
|
|
62
|
-
- step-06: 254 → 175 lines (gap format templates and remediation examples moved to data file)
|
|
63
|
-
|
|
64
|
-
---
|
|
65
|
-
|
|
66
|
-
## Frontmatter Validation
|
|
67
|
-
|
|
68
|
-
### Frontmatter Variables: PASS
|
|
69
|
-
|
|
70
|
-
All 21 frontmatter variables across 7 files are consumed in their step bodies.
|
|
71
|
-
|
|
72
|
-
| Step | Variables | All Used | Status |
|
|
73
|
-
|------|-----------|----------|--------|
|
|
74
|
-
| workflow.md | name, description, web_bundle, installed_path | Yes (metadata fields) | PASS |
|
|
75
|
-
| step-01 | nextStepFile, outputFile, templateFile, sidecarFile, skillsOutputFolder | Yes | PASS |
|
|
76
|
-
| step-02 | nextStepFile, outputFile | Yes | PASS |
|
|
77
|
-
| step-03 | nextStepFile, outputFile, scoringRulesFile | Yes | PASS |
|
|
78
|
-
| step-04 | nextStepFile, outputFile, outputFormatsFile | Yes | PASS |
|
|
79
|
-
| step-05 | nextStepFile, outputFile, scoringRulesFile | Yes | PASS |
|
|
80
|
-
| step-06 | outputFile, scoringRulesFile, outputFormatsFile | Yes | PASS |
|
|
81
|
-
|
|
82
|
-
### Path Format: PASS
|
|
83
|
-
|
|
84
|
-
- Step-to-step paths use `./filename.md` format (5/5)
|
|
85
|
-
- Parent-folder paths use `../filename.md` format (6/6)
|
|
86
|
-
- External paths use `{project-root}` prefix (2/2)
|
|
87
|
-
- No `{workflow_path}` variable present (7/7)
|
|
88
|
-
- All files have `name` + `description` fields (7/7)
|
|
89
|
-
|
|
90
|
-
---
|
|
91
|
-
|
|
92
|
-
## Critical Path Violations
|
|
93
|
-
|
|
94
|
-
### Hardcoded Paths in Step Bodies: PASS
|
|
95
|
-
|
|
96
|
-
All 6 step files scanned — zero violations. All path references use frontmatter variables.
|
|
97
|
-
|
|
98
|
-
### Workflow.md Bootstrap Paths: PASS (with 2 standard warnings)
|
|
99
|
-
|
|
100
|
-
| Line | Path | Context | Verdict |
|
|
101
|
-
|------|------|---------|---------|
|
|
102
|
-
| 50 | `{project-root}/_bmad/skf/config.yaml` | Config bootstrap | WARNING — standard entrypoint pattern |
|
|
103
|
-
| 57 | `./steps-c/step-01-init.md` | First step bootstrap | WARNING — standard entrypoint pattern |
|
|
104
|
-
|
|
105
|
-
Both are expected bootstrap patterns for workflow entrypoints.
|
|
106
|
-
|
|
107
|
-
### File Reference Chain: PASS
|
|
108
|
-
|
|
109
|
-
| Source | Reference | Target Exists |
|
|
110
|
-
|--------|-----------|---------------|
|
|
111
|
-
| step-01 → step-02 | `./step-02-detect-mode.md` | ✅ |
|
|
112
|
-
| step-02 → step-03 | `./step-03-coverage-check.md` | ✅ |
|
|
113
|
-
| step-03 → step-04 | `./step-04-coherence-check.md` | ✅ |
|
|
114
|
-
| step-04 → step-05 | `./step-05-score.md` | ✅ |
|
|
115
|
-
| step-05 → step-06 | `./step-06-report.md` | ✅ |
|
|
116
|
-
| step-06 | (final step — no nextStepFile) | ✅ |
|
|
117
|
-
| step-01 → template | `../templates/test-report-template.md` | ✅ |
|
|
118
|
-
| step-03, 05, 06 → scoring | `../data/scoring-rules.md` | ✅ |
|
|
119
|
-
| step-04, 06 → formats | `../data/output-section-formats.md` | ✅ |
|
|
120
|
-
|
|
121
|
-
---
|
|
122
|
-
|
|
123
|
-
## Menu Handling Validation
|
|
124
|
-
|
|
125
|
-
### Menu Compliance: PASS (71/71 checks)
|
|
126
|
-
|
|
127
|
-
| Step | Expected Pattern | Actual | Status |
|
|
128
|
-
|------|-----------------|--------|--------|
|
|
129
|
-
| step-01 | Auto-proceed (init) | Auto-proceed, no user menu | PASS |
|
|
130
|
-
| step-02 | Auto-proceed | Auto-proceed, no user menu | PASS |
|
|
131
|
-
| step-03 | Auto-proceed | Auto-proceed, no user menu | PASS |
|
|
132
|
-
| step-04 | Auto-proceed | Auto-proceed, no user menu | PASS |
|
|
133
|
-
| step-05 | Auto-proceed | Auto-proceed, no user menu | PASS |
|
|
134
|
-
| step-06 | C-only (final) | `[C] Finish` with halt-and-wait | PASS |
|
|
135
|
-
|
|
136
|
-
- ✅ No A/P options in any auto-proceed step
|
|
137
|
-
- ✅ Step-06 has "halt and wait" execution rule
|
|
138
|
-
- ✅ Step-06 redisplays menu for non-C input
|
|
139
|
-
- ✅ Step-06 correctly omits "loading next step" universal rule
|
|
140
|
-
|
|
141
|
-
---
|
|
142
|
-
|
|
143
|
-
## Step Type Validation
|
|
144
|
-
|
|
145
|
-
### Step Type Compliance: PASS (71/71 checks)
|
|
146
|
-
|
|
147
|
-
| Step | Expected Type | Match | All Sections Present | Status |
|
|
148
|
-
|------|--------------|-------|---------------------|--------|
|
|
149
|
-
| step-01 | Init + Input Discovery | ✅ | 6/6 | PASS |
|
|
150
|
-
| step-02 | Middle (Simple) | ✅ | 6/6 | PASS |
|
|
151
|
-
| step-03 | Validation Sequence | ✅ | 6/6 | PASS |
|
|
152
|
-
| step-04 | Validation Sequence (conditional) | ✅ | 6/6 | PASS |
|
|
153
|
-
| step-05 | Middle (Simple) | ✅ | 6/6 | PASS |
|
|
154
|
-
| step-06 | Final Step (C-only) | ✅ | 6/6 | PASS |
|
|
155
|
-
|
|
156
|
-
All steps have: STEP GOAL, MANDATORY EXECUTION RULES, EXECUTION PROTOCOLS, CONTEXT BOUNDARIES, MANDATORY SEQUENCE, SUCCESS/FAILURE METRICS.
|
|
157
|
-
|
|
158
|
-
**Remediation applied:** Fixed step-06 section numbering gap (was 1,2,3,4,6,7,8 → now 1,2,3,4,5,6,7).
|
|
159
|
-
|
|
160
|
-
---
|
|
161
|
-
|
|
162
|
-
## Output Format Validation
|
|
163
|
-
|
|
164
|
-
### Template Type: PASS
|
|
165
|
-
|
|
166
|
-
Template matches "structured" type with:
|
|
167
|
-
- YAML frontmatter with defined metadata fields
|
|
168
|
-
- Clear section headers: Test Summary, Coverage Analysis, Coherence Analysis, Completeness Score, Gap Report
|
|
169
|
-
- Skeletal structure (headers only) — each step fills its section
|
|
170
|
-
|
|
171
|
-
### Step-to-Output Mapping: PASS
|
|
172
|
-
|
|
173
|
-
| Step | Action | Output Section | outputFile in FM | Appends Before Next |
|
|
174
|
-
|------|--------|---------------|------------------|-------------------|
|
|
175
|
-
| 01 | Creates doc | (from template) | ✅ | ✅ |
|
|
176
|
-
| 02 | Appends | Test Summary | ✅ | ✅ |
|
|
177
|
-
| 03 | Appends | Coverage Analysis | ✅ | ✅ |
|
|
178
|
-
| 04 | Appends | Coherence Analysis | ✅ | ✅ |
|
|
179
|
-
| 05 | Appends | Completeness Score | ✅ | ✅ |
|
|
180
|
-
| 06 | Appends | Gap Report | ✅ | ✅ (final step) |
|
|
181
|
-
|
|
182
|
-
Every step has a CRITICAL STEP COMPLETION NOTE enforcing write-before-proceed.
|
|
183
|
-
|
|
184
|
-
### Final Polish Step: PASS (N/A)
|
|
185
|
-
|
|
186
|
-
Not required — structured template does not need editorial polish.
|
|
187
|
-
|
|
188
|
-
---
|
|
189
|
-
|
|
190
|
-
## Validation Design Check
|
|
191
|
-
|
|
192
|
-
### Validation Steps Required? N/A — PASS
|
|
193
|
-
|
|
194
|
-
test-skill IS itself a testing/validation workflow. It is the quality gate that tests skills. Adding validation steps (`steps-v/`) to a validation workflow would be circular.
|
|
195
|
-
|
|
196
|
-
- ✅ No `steps-v/` folder — correct for workflow purpose
|
|
197
|
-
- ✅ Workflow plan confirms: "Lifecycle Support: create-only" (no steps-e/ or steps-v/)
|
|
198
|
-
- ✅ The workflow's entire purpose is testing — every step IS a validation activity
|
|
199
|
-
|
|
200
|
-
---
|
|
201
|
-
|
|
202
|
-
## Instruction Style Check
|
|
203
|
-
|
|
204
|
-
### Domain Assessment: Prescriptive (correct)
|
|
205
|
-
|
|
206
|
-
test-skill is an audit/testing workflow (Ferris Audit mode) with deterministic analysis procedures, defined formulas, and exact comparison logic. Prescriptive style is appropriate.
|
|
207
|
-
|
|
208
|
-
### Per-Step Classification: PASS
|
|
209
|
-
|
|
210
|
-
| Step | Classification | Appropriate | Status |
|
|
211
|
-
|------|---------------|-------------|--------|
|
|
212
|
-
| step-01 | Prescriptive | ✅ Yes | PASS |
|
|
213
|
-
| step-02 | Prescriptive | ✅ Yes | PASS (1 warning) |
|
|
214
|
-
| step-03 | Prescriptive | ✅ Yes | PASS |
|
|
215
|
-
| step-04 | Prescriptive | ✅ Yes | PASS (1 warning) |
|
|
216
|
-
| step-05 | Prescriptive | ✅ Yes | PASS |
|
|
217
|
-
| step-06 | Prescriptive | ✅ Yes | PASS |
|
|
218
|
-
|
|
219
|
-
**Warnings:**
|
|
220
|
-
- ⚠️ step-02 lines 62-66: Skill type indicators mix definitive metadata checks with interpretive signals ("Self-contained API surface"). These are supplementary confirmation signals, not primary decision criteria (metadata field is checked first).
|
|
221
|
-
- ⚠️ step-04 lines 160-165: Integration pattern completeness criteria use qualitative language ("complete flow, not fragments"). Could benefit from more concrete criteria.
|
|
222
|
-
|
|
223
|
-
**Remediation applied:** Added coherence score aggregation formula to `scoring-rules.md` (was missing — agents would have had to guess how to combine reference validity and integration completeness).
|
|
224
|
-
|
|
225
|
-
---
|
|
226
|
-
|
|
227
|
-
## Collaborative Experience Check
|
|
228
|
-
|
|
229
|
-
### Interaction Pattern: Mostly Autonomous — PASS
|
|
230
|
-
|
|
231
|
-
| Step | Interaction | Quality |
|
|
232
|
-
|------|------------|---------|
|
|
233
|
-
| step-01 | Input discovery | Natural, progressive, single question — not a laundry list |
|
|
234
|
-
| step-02 | Autonomous | Clean auto-proceed, informative mode report |
|
|
235
|
-
| step-03 | Autonomous | Good progress reporting with coverage metrics |
|
|
236
|
-
| step-04 | Autonomous | Clean naive/contextual branching with proper fallbacks |
|
|
237
|
-
| step-05 | Autonomous | Excellent mathematical transparency ("show the math") |
|
|
238
|
-
| step-06 | Terminal menu | Clear results presentation, appropriate C-only menu |
|
|
239
|
-
|
|
240
|
-
### Workflow Progression: PASS
|
|
241
|
-
|
|
242
|
-
```
|
|
243
|
-
Init → Classify → Analyze Coverage → Analyze Coherence → Score → Report
|
|
244
|
-
```
|
|
245
|
-
|
|
246
|
-
Each step has explicit CONTEXT BOUNDARIES with FORBIDDEN markers preventing scope creep. Steps build logically on each other. Append-only document serves as state mechanism.
|
|
247
|
-
|
|
248
|
-
### Error Handling: Good
|
|
249
|
-
|
|
250
|
-
- step-01: Four-tier handling (hard stop for missing SKILL.md, warning for missing metadata.json, graceful default for missing forge-tier.yaml). Checks `preferences.yaml` for `tier_override` after loading forge tier.
|
|
251
|
-
- step-03/04: Subprocess fallback rules at multiple locations
|
|
252
|
-
- step-06: Zero-gaps clean pass path
|
|
253
|
-
|
|
254
|
-
**Warning:** step-03 has no explicit error handling for missing/inaccessible source files at `source_path`. Minor edge case.
|
|
255
|
-
|
|
256
|
-
### Overall Experience Quality: **Good**
|
|
257
|
-
|
|
258
|
-
---
|
|
259
|
-
|
|
260
|
-
## Subprocess Optimization Opportunities
|
|
261
|
-
|
|
262
|
-
### Usage Assessment: Good
|
|
263
|
-
|
|
264
|
-
| Step | Patterns Used | Fallback | Status |
|
|
265
|
-
|------|--------------|----------|--------|
|
|
266
|
-
| 01 | None (not needed) | Universal rule | PASS |
|
|
267
|
-
| 02 | None (not needed) | Universal rule | PASS |
|
|
268
|
-
| 03 | Pattern 2 (per-file AST analysis) | 3 locations | PASS |
|
|
269
|
-
| 04 | Pattern 1 (grep refs) + Pattern 2 (per-ref validation) | 3 locations | PASS |
|
|
270
|
-
| 05 | None (not needed) | Universal rule | PASS |
|
|
271
|
-
| 06 | None (not needed) | Universal rule | PASS |
|
|
272
|
-
|
|
273
|
-
Subprocess optimization correctly concentrated in steps 03 and 04 (heavy analysis). Lighter steps correctly avoid unnecessary overhead. All steps have subprocess fallback rules.
|
|
274
|
-
|
|
275
|
-
### Minor Opportunities:
|
|
276
|
-
|
|
277
|
-
- ⚠️ step-03 Quick tier: Could use Pattern 1 (single grep) for pre-scan before per-file analysis
|
|
278
|
-
- ⚠️ step-04: Per-reference validations are independent — could note Pattern 4 (parallel) opportunity
|
|
279
|
-
|
|
280
|
-
### Data Files: Well-Structured
|
|
281
|
-
|
|
282
|
-
- `scoring-rules.md` (72 lines): Clear tables, formulas, severity rules
|
|
283
|
-
- `output-section-formats.md` (97 lines): Self-contained format templates per section
|
|
284
|
-
|
|
285
|
-
---
|
|
286
|
-
|
|
287
|
-
## Cohesive Review
|
|
288
|
-
|
|
289
|
-
### Overall Assessment: **Good**
|
|
290
|
-
|
|
291
|
-
**Cohesiveness:** Excellent — all 6 steps build logically. Clean separation of concerns with each step having a single responsibility.
|
|
292
|
-
|
|
293
|
-
**Data Flow:** Good — append-only document building with proper frontmatter state tracking. One minor gap: step 02 references `{threshold}%` in Test Summary before step 05 loads it from scoring rules.
|
|
294
|
-
|
|
295
|
-
**Consistency:** Excellent — identical structure pattern across all steps (frontmatter → goal → rules → protocols → boundaries → sequence → metrics). Same role reinforcement ("Ferris Audit mode"), same emoji conventions, same subprocess fallback wording.
|
|
296
|
-
|
|
297
|
-
**Goal Achievement:** Good — would produce a useful, structured test report with pass/fail determination and actionable gap remediation. Tier-aware depth scaling ensures graceful degradation.
|
|
298
|
-
|
|
299
|
-
### Strengths:
|
|
300
|
-
|
|
301
|
-
1. Rigorous traceability (zero hallucination with file:line citations)
|
|
302
|
-
2. Four-tier depth scaling (Quick/Forge/Forge+/Deep)
|
|
303
|
-
3. Clean separation of concerns with explicit FORBIDDEN markers
|
|
304
|
-
4. Well-designed subprocess optimization with structured return formats
|
|
305
|
-
5. Actionable remediation quality rules with good/bad examples
|
|
306
|
-
6. Data files properly externalized for reuse
|
|
307
|
-
|
|
308
|
-
### Weaknesses:
|
|
309
|
-
|
|
310
|
-
1. Weight tables duplicated between `scoring-rules.md` and step-05 inline — maintenance risk
|
|
311
|
-
2. Batch mode referenced in workflow plan but not implemented in step files
|
|
312
|
-
3. Quick tier pass may give false confidence (name-matching only)
|
|
313
|
-
|
|
314
|
-
---
|
|
315
|
-
|
|
316
|
-
## Plan Quality Validation
|
|
317
|
-
|
|
318
|
-
### Plan Completeness: PASS
|
|
319
|
-
|
|
320
|
-
All 7 required sections present and substantive. Status: BUILD_COMPLETE with all 9 pipeline steps completed.
|
|
321
|
-
|
|
322
|
-
### Design-to-Implementation: PASS (1 warning fixed)
|
|
323
|
-
|
|
324
|
-
- ✅ All 6 planned steps have corresponding files with matching names
|
|
325
|
-
- ✅ Step descriptions align with plan goals
|
|
326
|
-
- ✅ File structure matches plan (fixed: added `output-section-formats.md` to plan)
|
|
327
|
-
|
|
328
|
-
### Consistency: PASS (1 warning)
|
|
329
|
-
|
|
330
|
-
- ✅ Step types match plan
|
|
331
|
-
- ✅ Menu patterns match plan
|
|
332
|
-
- ✅ Data flow matches plan
|
|
333
|
-
- ⚠️ Subprocess Pattern 4 (batch parallel) attributed to step 03 in plan table but is actually a workflow-level orchestration concern
|
|
334
|
-
|
|
335
|
-
### Plan Quality: PASS
|
|
336
|
-
|
|
337
|
-
- Conversion notes are insightful with genuine analysis
|
|
338
|
-
- Classification decisions well-reasoned with structure implications
|
|
339
|
-
- Requirements clear and complete with testable success criteria
|
|
340
|
-
- Design specification detailed enough to rebuild the workflow
|
|
341
|
-
|
|
342
|
-
---
|
|
343
|
-
|
|
344
|
-
## Summary
|
|
345
|
-
|
|
346
|
-
### Validation Complete: 2026-02-26
|
|
347
|
-
|
|
348
|
-
**Overall Status: PASS**
|
|
349
|
-
|
|
350
|
-
### Results Overview
|
|
351
|
-
|
|
352
|
-
| Validation Step | Result |
|
|
353
|
-
|----------------|--------|
|
|
354
|
-
| File Structure & Size | PASS (after remediation) |
|
|
355
|
-
| Frontmatter Validation | PASS |
|
|
356
|
-
| Critical Path Violations | PASS (2 standard warnings) |
|
|
357
|
-
| Menu Handling Validation | PASS (71/71) |
|
|
358
|
-
| Step Type Validation | PASS (71/71) |
|
|
359
|
-
| Output Format Validation | PASS |
|
|
360
|
-
| Validation Design Check | PASS (N/A) |
|
|
361
|
-
| Instruction Style Check | PASS (2 warnings) |
|
|
362
|
-
| Collaborative Experience Check | PASS |
|
|
363
|
-
| Subprocess Optimization | Good |
|
|
364
|
-
| Cohesive Review | Good |
|
|
365
|
-
| Plan Quality Validation | PASS (1 warning fixed) |
|
|
366
|
-
|
|
367
|
-
### Issues Summary
|
|
368
|
-
|
|
369
|
-
**Critical Issues:** 0
|
|
370
|
-
**Blocking Issues:** 0
|
|
371
|
-
**Warnings (non-blocking):** 7
|
|
372
|
-
|
|
373
|
-
1. step-02: Interpretive skill type indicators alongside definitive metadata check
|
|
374
|
-
2. step-04: Qualitative integration completeness criteria
|
|
375
|
-
3. step-03: No error handling for missing source files at source_path
|
|
376
|
-
4. step-05: Weight tables duplicated from scoring-rules.md
|
|
377
|
-
5. step-02: Threshold referenced before loaded by step-05
|
|
378
|
-
6. step-04: Could note Pattern 4 (parallel) for independent reference validations
|
|
379
|
-
7. Batch mode in plan but not in step files (unimplemented feature)
|
|
380
|
-
|
|
381
|
-
### Remediation Applied During Validation
|
|
382
|
-
|
|
383
|
-
1. Created `data/output-section-formats.md` — extracted templates to reduce step-04 and step-06 sizes
|
|
384
|
-
2. Fixed step-06 section numbering gap (4 → 6 became 4 → 5)
|
|
385
|
-
3. Added coherence score aggregation formula to `scoring-rules.md`
|
|
386
|
-
4. Updated plan file structure to include `output-section-formats.md`
|
|
387
|
-
|
|
388
|
-
### Recommendation
|
|
389
|
-
|
|
390
|
-
**Ready to use.** The test-skill workflow is well-architected with strong traceability, clean separation of concerns, and proper subprocess optimization. The 7 non-blocking warnings are minor quality improvements that can be addressed in a future iteration. The workflow will function correctly for single-skill testing in all three forge tiers.
|
|
@@ -1,292 +0,0 @@
|
|
|
1
|
-
---
|
|
2
|
-
conversionFrom: 'src/workflows/test-skill/test-skill.spec.md'
|
|
3
|
-
originalFormat: 'BMAD Workflow Specification (placeholder)'
|
|
4
|
-
stepsCompleted: ['step-00-conversion', 'step-02-classification', 'step-03-requirements', 'step-04-tools', 'step-05-plan-review', 'step-06-design', 'step-07-foundation', 'step-08-build-step-01', 'step-09-build-all-steps']
|
|
5
|
-
created: 2026-02-26
|
|
6
|
-
status: VALIDATED
|
|
7
|
-
approvedDate: 2026-02-26
|
|
8
|
-
validationDate: 2026-02-26
|
|
9
|
-
validationStatus: PASS
|
|
10
|
-
validationReport: './validation-report.md'
|
|
11
|
-
---
|
|
12
|
-
|
|
13
|
-
# Workflow Creation Plan
|
|
14
|
-
|
|
15
|
-
## Conversion Source
|
|
16
|
-
|
|
17
|
-
**Original Path:** src/workflows/test-skill/test-skill.spec.md
|
|
18
|
-
**Original Format:** BMAD Workflow Specification (placeholder — not yet implemented)
|
|
19
|
-
**Detected Structure:** Single specification document defining a 6-step create-only workflow
|
|
20
|
-
|
|
21
|
-
---
|
|
22
|
-
|
|
23
|
-
## Original Workflow Analysis
|
|
24
|
-
|
|
25
|
-
### Goal (from source)
|
|
26
|
-
|
|
27
|
-
Cognitive completeness verification — Verifies that a skill is complete enough to be useful to an AI agent. Supports two modes: naive (individual skill API surface coverage) and contextual (stack skill coherence validation). Produces a completeness score and gap report. Acts as a quality gate before export.
|
|
28
|
-
|
|
29
|
-
### Original Steps (Complete List)
|
|
30
|
-
|
|
31
|
-
**Step 1:** load-skill - Load skill and metadata
|
|
32
|
-
**Step 2:** detect-mode - Determine naive (individual) vs contextual (stack) testing mode
|
|
33
|
-
**Step 3:** coverage-check - Compare documented exports against source API surface
|
|
34
|
-
**Step 4:** coherence-check - Validate internal consistency (stack: references match SKILL.md)
|
|
35
|
-
**Step 5:** score - Calculate completeness score
|
|
36
|
-
**Step 6:** gap-report - Generate gap report with specific remediation suggestions
|
|
37
|
-
|
|
38
|
-
### Output / Deliverable
|
|
39
|
-
|
|
40
|
-
- Completeness score (pass/fail with percentage)
|
|
41
|
-
- Gap report with specific missing items and remediation suggestions
|
|
42
|
-
|
|
43
|
-
### Input Requirements
|
|
44
|
-
|
|
45
|
-
**Required:**
|
|
46
|
-
- Skill path (`{skills_output_folder}/{name}/`)
|
|
47
|
-
|
|
48
|
-
**Optional:**
|
|
49
|
-
- `--batch` flag with list of skills or directory
|
|
50
|
-
- Source path override
|
|
51
|
-
|
|
52
|
-
### Key Instructions to LLM
|
|
53
|
-
|
|
54
|
-
- Spec assigns **Ferris (Audit mode)** as the primary agent
|
|
55
|
-
- Create-only mode (steps-c/ only) — no edit or validate modal variants
|
|
56
|
-
- Document-producing workflow
|
|
57
|
-
- Must support `--batch` for testing multiple skills
|
|
58
|
-
- Quality gate role: determines if a skill is ready for export
|
|
59
|
-
- Two distinct analysis modes (naive vs contextual) require branching logic
|
|
60
|
-
|
|
61
|
-
---
|
|
62
|
-
|
|
63
|
-
## Conversion Notes
|
|
64
|
-
|
|
65
|
-
**What works well in original:**
|
|
66
|
-
- Clear step decomposition with well-defined goals
|
|
67
|
-
- Dual-mode architecture (naive/contextual) is well-scoped
|
|
68
|
-
- Input/output requirements are explicit
|
|
69
|
-
- Quality gate purpose is clear
|
|
70
|
-
- Correct lifecycle positioning: test-skill sits between create-skill and export-skill (Setup → Analyze → Brief → Create → **Test** → Export)
|
|
71
|
-
- Ferris Audit mode assignment is architecturally correct
|
|
72
|
-
|
|
73
|
-
**What needs improvement:**
|
|
74
|
-
- No step-level detail — only names and one-line goals exist
|
|
75
|
-
- No menu handling, user interaction points, or continuation flow defined
|
|
76
|
-
- No frontmatter configuration for steps
|
|
77
|
-
- No data files, templates, or output format specifications
|
|
78
|
-
- Batch mode mechanics are unspecified
|
|
79
|
-
- No forge tier awareness — coverage depth must scale with Quick/Forge/Forge+/Deep tiers
|
|
80
|
-
- No reference to forge-tier.yaml sidecar state for tier detection
|
|
81
|
-
- Zero hallucination principle not encoded — scoring must trace to AST citations
|
|
82
|
-
|
|
83
|
-
**Compliance gaps identified:**
|
|
84
|
-
- Missing: workflow.md entry point with proper frontmatter
|
|
85
|
-
- Missing: Step files with BMAD step-file architecture
|
|
86
|
-
- Missing: Frontmatter variables for configurable paths
|
|
87
|
-
- Missing: Menu handling and user interaction patterns
|
|
88
|
-
- Missing: State tracking via stepsCompleted
|
|
89
|
-
- Missing: Output document template/format specification
|
|
90
|
-
- Missing: Step processing rules and execution protocols
|
|
91
|
-
|
|
92
|
-
**Memory-informed design additions:**
|
|
93
|
-
- Tier-aware coverage: Quick=file/structure check, Forge=AST API surface, Deep=cross-repo coherence
|
|
94
|
-
- Batch mode: --batch with --continue for sidecar-checkpointed resume (matches create-skill pattern)
|
|
95
|
-
- Batch produces per-skill reports with optional consolidated summary
|
|
96
|
-
- Configurable pass/fail threshold (default overridable)
|
|
97
|
-
- Must read forge-tier.yaml from forger-sidecar to determine available analysis depth
|
|
98
|
-
|
|
99
|
-
---
|
|
100
|
-
|
|
101
|
-
## Classification Decisions
|
|
102
|
-
|
|
103
|
-
**Workflow Name:** test-skill
|
|
104
|
-
**Target Path:** {project-root}/_bmad/skf/workflows/test-skill
|
|
105
|
-
|
|
106
|
-
**4 Key Decisions:**
|
|
107
|
-
1. **Document Output:** true (completeness score + gap report)
|
|
108
|
-
2. **Module Affiliation:** SKF (Skill Forge)
|
|
109
|
-
3. **Session Type:** single-session
|
|
110
|
-
4. **Lifecycle Support:** create-only
|
|
111
|
-
|
|
112
|
-
**Structure Implications:**
|
|
113
|
-
- Only `steps-c/` folder needed (no steps-e/ or steps-v/)
|
|
114
|
-
- Standard init (no step-01b-continue.md needed)
|
|
115
|
-
- Free-form output template for test report document
|
|
116
|
-
- Access to SKF module variables (forge tier config, skills paths)
|
|
117
|
-
- Batch mode is a loop wrapper, not a multi-session concern
|
|
118
|
-
|
|
119
|
-
---
|
|
120
|
-
|
|
121
|
-
## Requirements
|
|
122
|
-
|
|
123
|
-
**Flow Structure:**
|
|
124
|
-
- Pattern: branching (by naive/contextual mode at step 02)
|
|
125
|
-
- Phases: Loading → Analysis → Scoring → Reporting
|
|
126
|
-
- Estimated steps: 6
|
|
127
|
-
- Branch point: Step 02 detect-mode determines naive vs contextual path
|
|
128
|
-
- Step 04 (coherence-check) is conditional — full execution in contextual mode, simplified/skipped in naive mode
|
|
129
|
-
|
|
130
|
-
**User Interaction:**
|
|
131
|
-
- Style: mostly autonomous (Ferris Audit mode — deterministic analysis)
|
|
132
|
-
- Decision points: none mid-flow; mode detection is automatic based on skill type
|
|
133
|
-
- Checkpoint frequency: entry (input validation) and exit (report delivery) only
|
|
134
|
-
|
|
135
|
-
**Inputs Required:**
|
|
136
|
-
- Required: skill path (`{skills_output_folder}/{name}/`), forge tier state (`forger-sidecar/forge-tier.yaml`)
|
|
137
|
-
- Optional: `--batch` flag with skill list or directory, `--continue` for interrupted batch resume, source path override, custom pass/fail threshold
|
|
138
|
-
- Prerequisites: setup-forge completed (tier detected), skill created via create-skill (SKILL.md exists)
|
|
139
|
-
|
|
140
|
-
**Output Specifications:**
|
|
141
|
-
- Type: document (structured test report)
|
|
142
|
-
- Format: structured (defined sections for machine-readability and batch comparison)
|
|
143
|
-
- Sections: Test Summary, Coverage Analysis, Coherence Analysis (contextual only), Completeness Score, Gap Report, Metadata
|
|
144
|
-
- Frequency: per-skill report; batch produces per-skill reports + optional consolidated summary
|
|
145
|
-
|
|
146
|
-
**Success Criteria:**
|
|
147
|
-
- Completeness score is accurate and traceable to code via AST citations (zero hallucination)
|
|
148
|
-
- Every gap has specific, actionable remediation suggestion
|
|
149
|
-
- Pass/fail threshold respected and clearly reported
|
|
150
|
-
- Naive mode correctly limits scope to API surface coverage
|
|
151
|
-
- Contextual mode validates cross-reference coherence for stack skills
|
|
152
|
-
- Batch mode produces consistent per-skill reports
|
|
153
|
-
- Report format parseable for downstream automation (export-skill can read it)
|
|
154
|
-
|
|
155
|
-
**Instruction Style:**
|
|
156
|
-
- Overall: prescriptive (audit/testing — exact procedures per check)
|
|
157
|
-
- Notes: Ferris Audit mode operates methodically; coverage check has specific comparison logic, coherence check has specific validation rules, scoring has defined formula
|
|
158
|
-
|
|
159
|
-
---
|
|
160
|
-
|
|
161
|
-
## Tools Configuration
|
|
162
|
-
|
|
163
|
-
**Core BMAD Tools:**
|
|
164
|
-
- **Party Mode:** excluded — deterministic audit, no creative exploration
|
|
165
|
-
- **Advanced Elicitation:** excluded — no facilitative questioning needed
|
|
166
|
-
- **Brainstorming:** excluded — no idea generation in testing/scoring workflow
|
|
167
|
-
|
|
168
|
-
**LLM Features:**
|
|
169
|
-
- **Web-Browsing:** excluded — all analysis is local (source code + SKILL.md)
|
|
170
|
-
- **File I/O:** included — read SKILL.md, metadata.json, forge-tier.yaml, source files; write test report
|
|
171
|
-
- **Sub-Agents:** included — coverage-check and coherence-check as parallel sub-agents (batch efficiency)
|
|
172
|
-
- **Sub-Processes:** included — batch mode parallel skill testing
|
|
173
|
-
|
|
174
|
-
**Memory:**
|
|
175
|
-
- Type: single-session (no continuation tracking)
|
|
176
|
-
- Sidecar: read-only access to forge-tier.yaml for tier detection; batch checkpointing for --continue
|
|
177
|
-
|
|
178
|
-
**External Integrations:**
|
|
179
|
-
- **ast-grep:** required for Forge/Deep tier — AST-backed API surface comparison in coverage-check
|
|
180
|
-
- **QMD:** optional for Deep tier — cross-reference knowledge search for coherence enrichment
|
|
181
|
-
- **gh CLI:** optional for Deep tier — cross-repo reference verification
|
|
182
|
-
|
|
183
|
-
**Installation Requirements:**
|
|
184
|
-
- All external tools managed by forge tier system (setup-forge detects availability)
|
|
185
|
-
- No additional installation — test-skill gracefully degrades based on detected tier
|
|
186
|
-
|
|
187
|
-
**Structure Preview:**
|
|
188
|
-
- Phase 1: Initialization — validate skill path, read forge tier, load metadata
|
|
189
|
-
- Phase 2: Mode Detection — determine naive vs contextual based on skill type
|
|
190
|
-
- Phase 3: Analysis — coverage-check (all tiers) + coherence-check (contextual only), depth scales with tier
|
|
191
|
-
- Phase 4: Scoring & Reporting — calculate score, apply threshold, generate gap report, write test report
|
|
192
|
-
|
|
193
|
-
---
|
|
194
|
-
|
|
195
|
-
## Workflow Design
|
|
196
|
-
|
|
197
|
-
### Step Sequence
|
|
198
|
-
|
|
199
|
-
| # | Name | Type | Menu | Goal |
|
|
200
|
-
|---|------|------|------|------|
|
|
201
|
-
| 01 | init | Init with Input Discovery | Auto-proceed | Discover skill, load forge tier, validate inputs |
|
|
202
|
-
| 02 | detect-mode | Middle (Simple) | Auto-proceed | Determine naive vs contextual from skill metadata |
|
|
203
|
-
| 03 | coverage-check | Validation Sequence | Auto-proceed | Compare documented exports against source API surface |
|
|
204
|
-
| 04 | coherence-check | Validation Sequence (conditional) | Auto-proceed | Validate references/coherence (contextual=full, naive=simplified) |
|
|
205
|
-
| 04b | external-validators | Middle (Simple) | Auto-proceed | Run external validation tools (skill-check, tessl) and capture scores |
|
|
206
|
-
| 05 | score | Middle (Simple) | Auto-proceed | Calculate completeness score from findings |
|
|
207
|
-
| 06 | report | Final Step | C only | Generate gap report, finalize document, recommend next workflow |
|
|
208
|
-
|
|
209
|
-
### Interaction Flow
|
|
210
|
-
|
|
211
|
-
```
|
|
212
|
-
User provides skill path → [01] init (auto) → [02] detect-mode (auto) →
|
|
213
|
-
[03] coverage-check (auto) → [04] coherence-check (auto) →
|
|
214
|
-
[04b] external-validators (auto) → [05] score (auto) →
|
|
215
|
-
[06] report → [C] finalize
|
|
216
|
-
```
|
|
217
|
-
|
|
218
|
-
### Data Flow
|
|
219
|
-
|
|
220
|
-
- Step 01: loads SKILL.md, metadata.json, forge-tier.yaml → creates output from template
|
|
221
|
-
- Step 02: reads metadata type → sets {testMode} naive|contextual → appends Test Summary
|
|
222
|
-
- Step 03: reads SKILL.md exports + source files → subprocess AST analysis → appends Coverage Analysis
|
|
223
|
-
- Step 04: reads SKILL.md references → conditional depth by mode → appends Coherence Analysis
|
|
224
|
-
- Step 04b: runs skill-check and tessl against skill directory → captures external scores → appends External Validation
|
|
225
|
-
- Step 05: reads coverage + coherence + external validation findings → calculates score → appends Completeness Score
|
|
226
|
-
- Step 06: reads all findings → generates remediation per gap → appends Gap Report → finalizes
|
|
227
|
-
|
|
228
|
-
### Subprocess Optimization
|
|
229
|
-
|
|
230
|
-
| Step | Pattern | What | Returns |
|
|
231
|
-
|------|---------|------|---------|
|
|
232
|
-
| 03 | Pattern 2 (deep analysis) | Per-source-file AST export comparison | {file, exports_found, exports_documented, missing, extra} |
|
|
233
|
-
| 03 | Pattern 4 (parallel) | Batch mode: parallel per-skill testing | Per-skill coverage findings |
|
|
234
|
-
| 04 | Pattern 1 (grep/regex) | Grep skill files for references | {references_found, files_checked, broken_refs} |
|
|
235
|
-
| 04 | Pattern 2 (deep analysis) | Per-reference coherence validation | {ref, target_exists, type_matches, signature_matches} |
|
|
236
|
-
|
|
237
|
-
Fallback: All subprocesses have main-thread fallback.
|
|
238
|
-
|
|
239
|
-
### File Structure
|
|
240
|
-
|
|
241
|
-
```
|
|
242
|
-
test-skill/
|
|
243
|
-
├── workflow.md
|
|
244
|
-
├── data/
|
|
245
|
-
│ ├── scoring-rules.md
|
|
246
|
-
│ └── output-section-formats.md
|
|
247
|
-
├── steps-c/
|
|
248
|
-
│ ├── step-01-init.md
|
|
249
|
-
│ ├── step-02-detect-mode.md
|
|
250
|
-
│ ├── step-03-coverage-check.md
|
|
251
|
-
│ ├── step-04-coherence-check.md
|
|
252
|
-
│ ├── step-04b-external-validators.md
|
|
253
|
-
│ ├── step-05-score.md
|
|
254
|
-
│ └── step-06-report.md
|
|
255
|
-
└── templates/
|
|
256
|
-
└── test-report-template.md
|
|
257
|
-
```
|
|
258
|
-
|
|
259
|
-
### Role & Persona
|
|
260
|
-
|
|
261
|
-
Ferris (Audit mode): methodical, precise, evidence-based. Zero hallucination — every finding cites AST evidence with file:line references. Tier-aware — states analysis depth in report. Non-judgmental — reports facts, suggests remediation.
|
|
262
|
-
|
|
263
|
-
### Workflow Chaining
|
|
264
|
-
|
|
265
|
-
- **Previous:** create-skill (required — SKILL.md must exist)
|
|
266
|
-
- **Next:** export-skill (if pass) or update-skill (if fail)
|
|
267
|
-
- **Input contract:** `{skills_output_folder}/{name}/SKILL.md` + `metadata.json` + `{sidecar_path}/forge-tier.yaml`
|
|
268
|
-
- **Output contract:** `{forge_data_folder}/{skill_name}/test-report-{skill_name}.md` with frontmatter: workflowType, testResult, score, nextWorkflow
|
|
269
|
-
|
|
270
|
-
### Output Template (Structured)
|
|
271
|
-
|
|
272
|
-
```markdown
|
|
273
|
-
---
|
|
274
|
-
workflowType: 'test-skill'
|
|
275
|
-
skillName: '{skill_name}'
|
|
276
|
-
testMode: '{naive|contextual}'
|
|
277
|
-
forgeTier: '{Quick|Forge|Forge+|Deep}'
|
|
278
|
-
testResult: '{pass|fail}'
|
|
279
|
-
score: '{N%}'
|
|
280
|
-
threshold: '{N%}'
|
|
281
|
-
testDate: '{date}'
|
|
282
|
-
nextWorkflow: '{export-skill|update-skill}'
|
|
283
|
-
---
|
|
284
|
-
|
|
285
|
-
# Test Report: {skill_name}
|
|
286
|
-
|
|
287
|
-
## Test Summary
|
|
288
|
-
## Coverage Analysis
|
|
289
|
-
## Coherence Analysis
|
|
290
|
-
## Completeness Score
|
|
291
|
-
## Gap Report
|
|
292
|
-
```
|