bmad-method-test-architecture-enterprise 1.23.2 → 1.23.3-next.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/marketplace.json +1 -1
- package/CHANGELOG.md +19 -0
- package/docs/explanation/step-file-architecture.md +1 -3
- package/docs/reference/configuration.md +8 -0
- package/eslint.config.mjs +0 -8
- package/package.json +1 -1
- package/src/workflows/testarch/bmad-teach-me-testing/steps-v/step-v-01-validate.md +18 -12
- package/src/workflows/testarch/bmad-testarch-atdd/steps-v/step-01-validate.md +16 -6
- package/src/workflows/testarch/bmad-testarch-automate/steps-v/step-01-validate.md +16 -6
- package/src/workflows/testarch/bmad-testarch-ci/steps-v/step-01-validate.md +17 -7
- package/src/workflows/testarch/bmad-testarch-framework/steps-v/step-01-validate.md +16 -6
- package/src/workflows/testarch/bmad-testarch-nfr/steps-v/step-01-validate.md +16 -6
- package/src/workflows/testarch/bmad-testarch-test-design/steps-v/step-01-validate.md +16 -6
- package/src/workflows/testarch/bmad-testarch-test-review/steps-v/step-01-validate.md +16 -6
- package/src/workflows/testarch/bmad-testarch-trace/steps-v/step-01-validate.md +16 -6
- package/test/README.md +1 -1
- package/test/test-installation-components.js +38 -0
- package/website/astro.config.mjs +1 -0
- package/src/workflows/testarch/bmad-teach-me-testing/workflow-plan-teach-me-testing.md +0 -950
- package/src/workflows/testarch/bmad-testarch-atdd/validation-report-20260127-095021.md +0 -73
- package/src/workflows/testarch/bmad-testarch-atdd/validation-report-20260127-102401.md +0 -116
- package/src/workflows/testarch/bmad-testarch-atdd/workflow-plan.md +0 -21
- package/src/workflows/testarch/bmad-testarch-automate/validation-report-20260127-095021.md +0 -72
- package/src/workflows/testarch/bmad-testarch-automate/validation-report-20260127-102401.md +0 -114
- package/src/workflows/testarch/bmad-testarch-automate/workflow-plan.md +0 -20
- package/src/workflows/testarch/bmad-testarch-ci/validation-report-20260127-095021.md +0 -72
- package/src/workflows/testarch/bmad-testarch-ci/validation-report-20260127-102401.md +0 -114
- package/src/workflows/testarch/bmad-testarch-ci/workflow-plan.md +0 -20
- package/src/workflows/testarch/bmad-testarch-framework/validation-report-20260127-095021.md +0 -73
- package/src/workflows/testarch/bmad-testarch-framework/validation-report-20260127-102401.md +0 -116
- package/src/workflows/testarch/bmad-testarch-framework/workflow-plan.md +0 -22
- package/src/workflows/testarch/bmad-testarch-nfr/validation-report-20260127-095021.md +0 -73
- package/src/workflows/testarch/bmad-testarch-nfr/validation-report-20260127-102401.md +0 -116
- package/src/workflows/testarch/bmad-testarch-nfr/workflow-plan.md +0 -19
- package/src/workflows/testarch/bmad-testarch-test-design/validation-report-20260127-095021.md +0 -73
- package/src/workflows/testarch/bmad-testarch-test-design/validation-report-20260127-102401.md +0 -116
- package/src/workflows/testarch/bmad-testarch-test-design/workflow-plan.md +0 -23
- package/src/workflows/testarch/bmad-testarch-test-review/validation-report-20260127-095021.md +0 -72
- package/src/workflows/testarch/bmad-testarch-test-review/validation-report-20260127-102401.md +0 -114
- package/src/workflows/testarch/bmad-testarch-test-review/workflow-plan.md +0 -18
- package/src/workflows/testarch/bmad-testarch-trace/validation-report-20260127-095021.md +0 -73
- package/src/workflows/testarch/bmad-testarch-trace/validation-report-20260127-102401.md +0 -116
- package/src/workflows/testarch/bmad-testarch-trace/workflow-plan.md +0 -28
- package/test/unit-test-schema.js +0 -133
- package/test/validate-agent-schema.js +0 -110
- package/tools/verify-paths.js +0 -100
|
@@ -31,7 +31,7 @@
|
|
|
31
31
|
"name": "bmad-method-test-architecture-enterprise",
|
|
32
32
|
"source": "./",
|
|
33
33
|
"description": "Master Test Architect module for quality strategy, test automation, CI/CD quality gates, and structured testing education. Part of the BMad Method ecosystem.",
|
|
34
|
-
"version": "1.23.
|
|
34
|
+
"version": "1.23.3-next.1",
|
|
35
35
|
"author": {
|
|
36
36
|
"name": "Murat K Ozcan (TEA Creator) & Brian (BMad) Madison"
|
|
37
37
|
},
|
package/CHANGELOG.md
CHANGED
|
@@ -7,6 +7,25 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
7
7
|
|
|
8
8
|
## [Unreleased]
|
|
9
9
|
|
|
10
|
+
### Fixed
|
|
11
|
+
|
|
12
|
+
- Validate mode now writes a scope-specific, timestamped report for each run instead of reusing one fixed filename. Artifact validation reports record the exact files checked, and every validate workflow atomically reserves its report through exclusive creation with collision retry, so sequential and parallel QA runs retain independent history (#140).
|
|
13
|
+
|
|
14
|
+
## [1.23.3] - 2026-08-19
|
|
15
|
+
|
|
16
|
+
### Removed
|
|
17
|
+
|
|
18
|
+
- Stale BMad Builder run artifacts from the skills migration. `e295715` carried 16 `validation-report-20260127-*.md` files, two per workflow across the eight `bmad-testarch-*` workflows, recording a 2026-01-27 validation run. Every one of them asserted `workflow.md present: YES` for a file that no longer exists in any workflow (it became `SKILL.md` in that same migration), and every step-file count and line count in them disagreed with the tree they shipped alongside. Each workflow directory is a published skill root in `.claude-plugin/marketplace.json`, so these were shipping to consumers as skill payload, where a wrong file listing is worse than no file listing.
|
|
19
|
+
- The nine authoring-time workflow plans: `workflow-plan.md` in each of the eight `bmad-testarch-*` workflows, plus `workflow-plan-teach-me-testing.md`. Nothing in `src/`, `test/`, `tools/`, `cli/`, or the docs read them; the only inbound references were from the validation reports removed above. Seven of the eight listed a step set that no longer matched their own directory, and the `teach-me-testing` one was a 950-line build-session log carrying `status: FOUNDATION_COMPLETE` frontmatter and a `cp -r` deployment instruction pointing at an external project root. Like the reports, they shipped inside a skill root.
|
|
20
|
+
- Three orphaned scripts from the TEA migration, each with zero inbound references from `package.json`, CI, or the pre-commit hook. `test/validate-agent-schema.js` was a stale copy of `tools/validate-agent-schema.js` whose own usage line named the `tools/` path; it globbed the abandoned `src/{core,modules/*}/agents/` layout and printed `No agent files found` when run. `test/unit-test-schema.js` failed one of its own three assertions and exited 0 regardless, so wiring it up would have gated nothing. `tools/verify-paths.js` was a one-off checker for leftover BMM references, and the migration it guarded is finished.
|
|
21
|
+
- Dead ignore entries that never matched anything in this repository's history: `test/template-test-generator/**` and its two sibling globs, `tools/template-test-generator/test-scenarios/**`, `test-project-install/**`, `sample-project/**`, `src/modules/*/sub-modules/**`, and `.bundler-temp/**` from `eslint.config.mjs`; `test-output/*` from `.gitignore`, along with its redundant `build/*.txt` entry, which `build/` already covers.
|
|
22
|
+
|
|
23
|
+
### Fixed
|
|
24
|
+
|
|
25
|
+
- `docs/explanation/step-file-architecture.md` no longer claims the workflows carry committed validation reports "each scoring 100%". No score appears anywhere in the 16 reports the sentence described, so the claim was unverifiable against its own evidence before those reports were removed. The section now states what validation checks and that its output is a point-in-time reading of the working tree rather than a committed artifact.
|
|
26
|
+
- `docs/explanation/eval-quality-roadmap.md` is reachable from the documentation site. It was the only page in `docs/explanation/` missing from the Starlight sidebar in `website/astro.config.mjs`, so it built and published but could only be found through the direct link in `README.md`.
|
|
27
|
+
- `test/README.md` names the validator the suite actually loads. It cited `tools/schema/agent.js`, while `test/test-agent-schema.js` requires the byte-identical copy at `test/schema/agent.js`.
|
|
28
|
+
|
|
10
29
|
## [1.23.1] - 2026-08-16
|
|
11
30
|
|
|
12
31
|
### Changed
|
|
@@ -194,9 +194,7 @@ Users do not need to know any of this to run a workflow. What they see is consis
|
|
|
194
194
|
|
|
195
195
|
## Validation
|
|
196
196
|
|
|
197
|
-
|
|
198
|
-
|
|
199
|
-
Validation checks for granular instructions, explicit exit conditions, context injection in every step, strict action boundaries, and subagent support where the workflow supports it.
|
|
197
|
+
Every workflow is validated with BMad Builder, which checks for granular instructions, explicit exit conditions, context injection in every step, strict action boundaries, and subagent support where the workflow supports it. Validation runs against the working tree at the time it is invoked, so its output is a point-in-time reading rather than a durable artifact; the reports are not committed. Re-run BMad Builder validation after editing a step file, and read the result from that run.
|
|
200
198
|
|
|
201
199
|
All nine workflows have been exercised against real projects: `teach-me-testing` across a multi-session flow with persisted progress, `test-design` against a real story and epic, `automate` against real codebases, `atdd` for the red phase with failing tests confirmed, `test-review` against known good and bad suites, `nfr-assess` against a complex system, `trace` for both the coverage matrix and the gate decision, `framework` for Playwright and Cypress scaffolds, and `ci` for GitHub Actions and GitLab CI generation.
|
|
202
200
|
|
|
@@ -474,6 +474,14 @@ Paths are relative to `{test_artifacts}` unless noted. Deliverables are declared
|
|
|
474
474
|
|
|
475
475
|
`trace` also reads an optional input it never writes: `live-verification-results.json`. Any producer may write it (an agent, a shell script, a CI job, or a person recording an outcome by hand). See [Live Verification Results](/docs/reference/live-verification-results.md) for the contract.
|
|
476
476
|
|
|
477
|
+
### Validation Report History
|
|
478
|
+
|
|
479
|
+
Validate mode preserves every report as a separate artifact. The eight artifact-producing workflows write `{workflow}-validation-report-{validation_scope}-{run_timestamp}.md` under `{test_artifacts}`. The workflow identifier is `atdd`, `automate`, `ci`, `framework`, `nfr-assess`, `test-design`, `test-review`, or `trace`.
|
|
480
|
+
|
|
481
|
+
`validation_scope` identifies what was checked, such as `story-1-2`, `epic-9`, `system`, or `pull-request-123`. `run_timestamp` is the UTC start time with milliseconds in `YYYYMMDDTHHmmssSSSZ` format. Each report also records the exact project-relative paths of its validated artifacts. Validate mode atomically reserves the resolved path with exclusive creation. A collision produces a fresh timestamp and retry, so two concurrent runs cannot claim the same report.
|
|
482
|
+
|
|
483
|
+
The `teach-me-testing` workflow validates its own workflow definition rather than a selected output scope. Its reports use `workflow-validation/teach-me-testing-validation-{run_timestamp}.md` under `{test_artifacts}` and follow the same no-overwrite rule.
|
|
484
|
+
|
|
477
485
|
---
|
|
478
486
|
|
|
479
487
|
## Environment Variables
|
package/eslint.config.mjs
CHANGED
|
@@ -11,9 +11,6 @@ export default [
|
|
|
11
11
|
'dist/**',
|
|
12
12
|
'coverage/**',
|
|
13
13
|
'**/*.min.js',
|
|
14
|
-
'test/template-test-generator/**',
|
|
15
|
-
'test/template-test-generator/**/*.js',
|
|
16
|
-
'test/template-test-generator/**/*.md',
|
|
17
14
|
'test/fixtures/**',
|
|
18
15
|
'test/fixtures/**/*.yaml',
|
|
19
16
|
'_bmad/**',
|
|
@@ -31,11 +28,6 @@ export default [
|
|
|
31
28
|
'.agentvibes/**',
|
|
32
29
|
'.kiro/**',
|
|
33
30
|
'.roo/**',
|
|
34
|
-
'test-project-install/**',
|
|
35
|
-
'sample-project/**',
|
|
36
|
-
'tools/template-test-generator/test-scenarios/**',
|
|
37
|
-
'src/modules/*/sub-modules/**',
|
|
38
|
-
'.bundler-temp/**',
|
|
39
31
|
],
|
|
40
32
|
},
|
|
41
33
|
|
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"$schema": "https://json.schemastore.org/package.json",
|
|
3
3
|
"name": "bmad-method-test-architecture-enterprise",
|
|
4
|
-
"version": "1.23.
|
|
4
|
+
"version": "1.23.3-next.1",
|
|
5
5
|
"description": "Master Test Architect for quality strategy, test automation, and release gates",
|
|
6
6
|
"keywords": [
|
|
7
7
|
"bmad",
|
|
@@ -4,7 +4,7 @@ description: 'Validate teach-me-testing workflow quality against BMAD standards'
|
|
|
4
4
|
|
|
5
5
|
workflowPath: '{skill-root}'
|
|
6
6
|
checklistFile: '{skill-root}/checklist.md'
|
|
7
|
-
validationReport: '{test_artifacts}/workflow-validation/teach-me-testing-validation-{
|
|
7
|
+
validationReport: '{test_artifacts}/workflow-validation/teach-me-testing-validation-{run_timestamp}.md'
|
|
8
8
|
---
|
|
9
9
|
|
|
10
10
|
# Validate Step 1: Quality Validation
|
|
@@ -30,6 +30,7 @@ To systematically validate the teach-me-testing workflow against BMAD quality st
|
|
|
30
30
|
|
|
31
31
|
- 🎯 Focus on comprehensive validation
|
|
32
32
|
- 🚫 FORBIDDEN to skip any checks
|
|
33
|
+
- 🚫 Never overwrite an existing validation report
|
|
33
34
|
- 💬 Report findings clearly
|
|
34
35
|
|
|
35
36
|
## EXECUTION PROTOCOLS:
|
|
@@ -40,7 +41,11 @@ To systematically validate the teach-me-testing workflow against BMAD quality st
|
|
|
40
41
|
|
|
41
42
|
## MANDATORY SEQUENCE
|
|
42
43
|
|
|
43
|
-
### 1.
|
|
44
|
+
### 1. Resolve Unique Report Path
|
|
45
|
+
|
|
46
|
+
Set `run_timestamp` to the current UTC time with milliseconds in `YYYYMMDDTHHmmssSSSZ` format and resolve `{validationReport}`. Atomically reserve that path using an exclusive-create operation that fails if the file already exists. A separate existence check followed by a normal write is forbidden. On collision, generate a fresh timestamp, resolve a new path, and retry exclusive creation until it succeeds. Initialize the reserved file with `workflow: teach-me-testing`, `run_timestamp`, and `status: IN_PROGRESS`. This run may update only the file it reserved. If the workflow stops, leave that reservation in place. Never delete, truncate, or reuse a report from another run. Always refuse to overwrite prior validation history.
|
|
47
|
+
|
|
48
|
+
### 2. Validation Start
|
|
44
49
|
|
|
45
50
|
"**Validating Workflow: teach-me-testing**
|
|
46
51
|
|
|
@@ -59,7 +64,7 @@ This will validate:
|
|
|
59
64
|
|
|
60
65
|
**Starting validation...**"
|
|
61
66
|
|
|
62
|
-
###
|
|
67
|
+
### 3. Foundation Structure Validation
|
|
63
68
|
|
|
64
69
|
**Check:**
|
|
65
70
|
|
|
@@ -72,7 +77,7 @@ This will validate:
|
|
|
72
77
|
|
|
73
78
|
Report findings: Pass/Fail for each check.
|
|
74
79
|
|
|
75
|
-
###
|
|
80
|
+
### 4. Template Validation
|
|
76
81
|
|
|
77
82
|
**Check templates/:**
|
|
78
83
|
|
|
@@ -85,7 +90,7 @@ Report findings: Pass/Fail for each check.
|
|
|
85
90
|
|
|
86
91
|
Report findings.
|
|
87
92
|
|
|
88
|
-
###
|
|
93
|
+
### 5. Step File Validation (CREATE Mode)
|
|
89
94
|
|
|
90
95
|
**For each of 12 steps in steps-c/:**
|
|
91
96
|
|
|
@@ -100,7 +105,7 @@ Report findings.
|
|
|
100
105
|
|
|
101
106
|
Report findings per step.
|
|
102
107
|
|
|
103
|
-
###
|
|
108
|
+
### 6. Data File Validation
|
|
104
109
|
|
|
105
110
|
**Check data/:**
|
|
106
111
|
|
|
@@ -112,7 +117,7 @@ Report findings per step.
|
|
|
112
117
|
|
|
113
118
|
Report findings.
|
|
114
119
|
|
|
115
|
-
###
|
|
120
|
+
### 7. Content Quality Validation
|
|
116
121
|
|
|
117
122
|
**Check session steps:**
|
|
118
123
|
|
|
@@ -125,7 +130,7 @@ Report findings.
|
|
|
125
130
|
|
|
126
131
|
Report findings.
|
|
127
132
|
|
|
128
|
-
###
|
|
133
|
+
### 8. State Management Validation
|
|
129
134
|
|
|
130
135
|
**Check continuable workflow features:**
|
|
131
136
|
|
|
@@ -138,7 +143,7 @@ Report findings.
|
|
|
138
143
|
|
|
139
144
|
Report findings.
|
|
140
145
|
|
|
141
|
-
###
|
|
146
|
+
### 9. User Experience Validation
|
|
142
147
|
|
|
143
148
|
**Check UX:**
|
|
144
149
|
|
|
@@ -151,14 +156,15 @@ Report findings.
|
|
|
151
156
|
|
|
152
157
|
Report findings.
|
|
153
158
|
|
|
154
|
-
###
|
|
159
|
+
### 10. Generate Validation Report
|
|
155
160
|
|
|
156
|
-
|
|
161
|
+
Replace the `IN_PROGRESS` body in this run's reserved {validationReport} with:
|
|
157
162
|
|
|
158
163
|
```markdown
|
|
159
164
|
---
|
|
160
165
|
workflow: teach-me-testing
|
|
161
166
|
validation_date: { current_date }
|
|
167
|
+
run_timestamp: { run_timestamp }
|
|
162
168
|
validator: TEA Validation Workflow
|
|
163
169
|
overall_status: PASS / FAIL / PASS_WITH_WARNINGS
|
|
164
170
|
---
|
|
@@ -231,7 +237,7 @@ overall_status: PASS / FAIL / PASS_WITH_WARNINGS
|
|
|
231
237
|
**Status:** {READY_FOR_PRODUCTION / NEEDS_FIXES / PASS_WITH_MINOR_ISSUES}
|
|
232
238
|
```
|
|
233
239
|
|
|
234
|
-
###
|
|
240
|
+
### 11. Display Results
|
|
235
241
|
|
|
236
242
|
"**Validation Complete!**
|
|
237
243
|
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: 'step-01-validate'
|
|
3
3
|
description: 'Validate workflow outputs against checklist'
|
|
4
|
-
outputFile: '{test_artifacts}/atdd-validation-report.md'
|
|
4
|
+
outputFile: '{test_artifacts}/atdd-validation-report-{validation_scope}-{run_timestamp}.md'
|
|
5
5
|
validationChecklist: '{skill-root}/checklist.md'
|
|
6
6
|
---
|
|
7
7
|
|
|
@@ -26,6 +26,7 @@ Validate outputs using the workflow checklist and record findings.
|
|
|
26
26
|
|
|
27
27
|
- 🎯 Validate against `{validationChecklist}`
|
|
28
28
|
- 🚫 Do not skip checks
|
|
29
|
+
- 🚫 Never overwrite an existing validation report
|
|
29
30
|
|
|
30
31
|
## EXECUTION PROTOCOLS:
|
|
31
32
|
|
|
@@ -34,7 +35,7 @@ Validate outputs using the workflow checklist and record findings.
|
|
|
34
35
|
|
|
35
36
|
## CONTEXT BOUNDARIES:
|
|
36
37
|
|
|
37
|
-
- Available context: workflow outputs and checklist
|
|
38
|
+
- Available context: user-selected workflow outputs and checklist
|
|
38
39
|
- Focus: validation only
|
|
39
40
|
- Limits: do not modify outputs in this step
|
|
40
41
|
|
|
@@ -42,17 +43,25 @@ Validate outputs using the workflow checklist and record findings.
|
|
|
42
43
|
|
|
43
44
|
**CRITICAL:** Follow this sequence exactly.
|
|
44
45
|
|
|
45
|
-
### 1.
|
|
46
|
+
### 1. Select Scope and Resolve Report Path
|
|
47
|
+
|
|
48
|
+
Use the artifact paths the user supplied with the Validate request. If none were supplied, list the likely outputs for this workflow and ask which exact file or files to validate. When several candidates exist, do not guess.
|
|
49
|
+
|
|
50
|
+
Read the selected artifacts. Derive `validation_scope` from their shared story, epic, system, pull request, or other meaningful scope. Use an artifact basename without its extension when no broader scope is available. Normalize the value to lowercase ASCII with only letters, numbers, and single hyphens. Remove leading and trailing hyphens. Ask for a short scope label if normalization leaves an empty value.
|
|
51
|
+
|
|
52
|
+
Set `run_timestamp` to the current UTC time with milliseconds in `YYYYMMDDTHHmmssSSSZ` format and resolve `{outputFile}` with both values. Atomically reserve that path using an exclusive-create operation that fails if the file already exists. A separate existence check followed by a normal write is forbidden. On collision, generate a fresh timestamp, resolve a new path, and retry exclusive creation until it succeeds. Initialize the reserved file with `validation_scope`, `run_timestamp`, `validated_artifacts`, and `status: IN_PROGRESS`. This run may update only the file it reserved. If the workflow stops, leave that reservation in place. Never delete, truncate, or reuse a report from another run. Always refuse to overwrite prior validation history.
|
|
53
|
+
|
|
54
|
+
### 2. Load Checklist
|
|
46
55
|
|
|
47
56
|
Read `{validationChecklist}` and list all criteria.
|
|
48
57
|
|
|
49
|
-
###
|
|
58
|
+
### 3. Validate Outputs
|
|
50
59
|
|
|
51
60
|
Evaluate outputs against each checklist item.
|
|
52
61
|
|
|
53
|
-
###
|
|
62
|
+
### 4. Write Report
|
|
54
63
|
|
|
55
|
-
|
|
64
|
+
Replace the `IN_PROGRESS` body in this run's reserved `{outputFile}` with the final validation report. Include PASS/WARN/FAIL per section plus the original `validation_scope`, `run_timestamp`, and `validated_artifacts` metadata. Record every selected artifact using its exact project-relative path.
|
|
56
65
|
|
|
57
66
|
## 🚨 SYSTEM SUCCESS/FAILURE METRICS:
|
|
58
67
|
|
|
@@ -60,6 +69,7 @@ Write a validation report to `{outputFile}` with PASS/WARN/FAIL per section.
|
|
|
60
69
|
|
|
61
70
|
- Validation report written
|
|
62
71
|
- All checklist items evaluated
|
|
72
|
+
- All selected artifacts recorded in the report
|
|
63
73
|
|
|
64
74
|
### ❌ SYSTEM FAILURE:
|
|
65
75
|
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: 'step-01-validate'
|
|
3
3
|
description: 'Validate workflow outputs against checklist'
|
|
4
|
-
outputFile: '{test_artifacts}/automate-validation-report.md'
|
|
4
|
+
outputFile: '{test_artifacts}/automate-validation-report-{validation_scope}-{run_timestamp}.md'
|
|
5
5
|
validationChecklist: '{skill-root}/checklist.md'
|
|
6
6
|
---
|
|
7
7
|
|
|
@@ -26,6 +26,7 @@ Validate outputs using the workflow checklist and record findings.
|
|
|
26
26
|
|
|
27
27
|
- 🎯 Validate against `{validationChecklist}`
|
|
28
28
|
- 🚫 Do not skip checks
|
|
29
|
+
- 🚫 Never overwrite an existing validation report
|
|
29
30
|
|
|
30
31
|
## EXECUTION PROTOCOLS:
|
|
31
32
|
|
|
@@ -34,7 +35,7 @@ Validate outputs using the workflow checklist and record findings.
|
|
|
34
35
|
|
|
35
36
|
## CONTEXT BOUNDARIES:
|
|
36
37
|
|
|
37
|
-
- Available context: workflow outputs and checklist
|
|
38
|
+
- Available context: user-selected workflow outputs and checklist
|
|
38
39
|
- Focus: validation only
|
|
39
40
|
- Limits: do not modify outputs in this step
|
|
40
41
|
|
|
@@ -42,17 +43,25 @@ Validate outputs using the workflow checklist and record findings.
|
|
|
42
43
|
|
|
43
44
|
**CRITICAL:** Follow this sequence exactly.
|
|
44
45
|
|
|
45
|
-
### 1.
|
|
46
|
+
### 1. Select Scope and Resolve Report Path
|
|
47
|
+
|
|
48
|
+
Use the artifact paths the user supplied with the Validate request. If none were supplied, list the likely outputs for this workflow and ask which exact file or files to validate. When several candidates exist, do not guess.
|
|
49
|
+
|
|
50
|
+
Read the selected artifacts. Derive `validation_scope` from their shared story, epic, system, pull request, or other meaningful scope. Use an artifact basename without its extension when no broader scope is available. Normalize the value to lowercase ASCII with only letters, numbers, and single hyphens. Remove leading and trailing hyphens. Ask for a short scope label if normalization leaves an empty value.
|
|
51
|
+
|
|
52
|
+
Set `run_timestamp` to the current UTC time with milliseconds in `YYYYMMDDTHHmmssSSSZ` format and resolve `{outputFile}` with both values. Atomically reserve that path using an exclusive-create operation that fails if the file already exists. A separate existence check followed by a normal write is forbidden. On collision, generate a fresh timestamp, resolve a new path, and retry exclusive creation until it succeeds. Initialize the reserved file with `validation_scope`, `run_timestamp`, `validated_artifacts`, and `status: IN_PROGRESS`. This run may update only the file it reserved. If the workflow stops, leave that reservation in place. Never delete, truncate, or reuse a report from another run. Always refuse to overwrite prior validation history.
|
|
53
|
+
|
|
54
|
+
### 2. Load Checklist
|
|
46
55
|
|
|
47
56
|
Read `{validationChecklist}` and list all criteria.
|
|
48
57
|
|
|
49
|
-
###
|
|
58
|
+
### 3. Validate Outputs
|
|
50
59
|
|
|
51
60
|
Evaluate outputs against each checklist item.
|
|
52
61
|
|
|
53
|
-
###
|
|
62
|
+
### 4. Write Report
|
|
54
63
|
|
|
55
|
-
|
|
64
|
+
Replace the `IN_PROGRESS` body in this run's reserved `{outputFile}` with the final validation report. Include PASS/WARN/FAIL per section plus the original `validation_scope`, `run_timestamp`, and `validated_artifacts` metadata. Record every selected artifact using its exact project-relative path.
|
|
56
65
|
|
|
57
66
|
## 🚨 SYSTEM SUCCESS/FAILURE METRICS:
|
|
58
67
|
|
|
@@ -60,6 +69,7 @@ Write a validation report to `{outputFile}` with PASS/WARN/FAIL per section.
|
|
|
60
69
|
|
|
61
70
|
- Validation report written
|
|
62
71
|
- All checklist items evaluated
|
|
72
|
+
- All selected artifacts recorded in the report
|
|
63
73
|
|
|
64
74
|
### ❌ SYSTEM FAILURE:
|
|
65
75
|
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: 'step-01-validate'
|
|
3
3
|
description: 'Validate workflow outputs against checklist'
|
|
4
|
-
outputFile: '{test_artifacts}/ci-validation-report.md'
|
|
4
|
+
outputFile: '{test_artifacts}/ci-validation-report-{validation_scope}-{run_timestamp}.md'
|
|
5
5
|
validationChecklist: '{skill-root}/checklist.md'
|
|
6
6
|
---
|
|
7
7
|
|
|
@@ -26,6 +26,7 @@ Validate outputs using the workflow checklist and record findings.
|
|
|
26
26
|
|
|
27
27
|
- 🎯 Validate against `{validationChecklist}`
|
|
28
28
|
- 🚫 Do not skip checks
|
|
29
|
+
- 🚫 Never overwrite an existing validation report
|
|
29
30
|
|
|
30
31
|
## EXECUTION PROTOCOLS:
|
|
31
32
|
|
|
@@ -34,7 +35,7 @@ Validate outputs using the workflow checklist and record findings.
|
|
|
34
35
|
|
|
35
36
|
## CONTEXT BOUNDARIES:
|
|
36
37
|
|
|
37
|
-
- Available context: workflow outputs and checklist
|
|
38
|
+
- Available context: user-selected workflow outputs and checklist
|
|
38
39
|
- Focus: validation only
|
|
39
40
|
- Limits: do not modify outputs in this step
|
|
40
41
|
|
|
@@ -42,15 +43,23 @@ Validate outputs using the workflow checklist and record findings.
|
|
|
42
43
|
|
|
43
44
|
**CRITICAL:** Follow this sequence exactly.
|
|
44
45
|
|
|
45
|
-
### 1.
|
|
46
|
+
### 1. Select Scope and Resolve Report Path
|
|
47
|
+
|
|
48
|
+
Use the artifact paths the user supplied with the Validate request. If none were supplied, list the likely outputs for this workflow and ask which exact file or files to validate. When several candidates exist, do not guess.
|
|
49
|
+
|
|
50
|
+
Read the selected artifacts. Derive `validation_scope` from their shared story, epic, system, pull request, or other meaningful scope. Use an artifact basename without its extension when no broader scope is available. Normalize the value to lowercase ASCII with only letters, numbers, and single hyphens. Remove leading and trailing hyphens. Ask for a short scope label if normalization leaves an empty value.
|
|
51
|
+
|
|
52
|
+
Set `run_timestamp` to the current UTC time with milliseconds in `YYYYMMDDTHHmmssSSSZ` format and resolve `{outputFile}` with both values. Atomically reserve that path using an exclusive-create operation that fails if the file already exists. A separate existence check followed by a normal write is forbidden. On collision, generate a fresh timestamp, resolve a new path, and retry exclusive creation until it succeeds. Initialize the reserved file with `validation_scope`, `run_timestamp`, `validated_artifacts`, and `status: IN_PROGRESS`. This run may update only the file it reserved. If the workflow stops, leave that reservation in place. Never delete, truncate, or reuse a report from another run. Always refuse to overwrite prior validation history.
|
|
53
|
+
|
|
54
|
+
### 2. Load Checklist
|
|
46
55
|
|
|
47
56
|
Read `{validationChecklist}` and list all criteria.
|
|
48
57
|
|
|
49
|
-
###
|
|
58
|
+
### 3. Validate Outputs
|
|
50
59
|
|
|
51
60
|
Evaluate outputs against each checklist item.
|
|
52
61
|
|
|
53
|
-
###
|
|
62
|
+
### 3a. Script Injection Scan
|
|
54
63
|
|
|
55
64
|
Scan all generated YAML workflow files for unsafe interpolation patterns inside `run:` blocks.
|
|
56
65
|
|
|
@@ -64,9 +73,9 @@ Scan all generated YAML workflow files for unsafe interpolation patterns inside
|
|
|
64
73
|
|
|
65
74
|
**Safe patterns to ignore** (exempt from flagging): `${{ steps.*.outputs.* }}`, `${{ matrix.* }}`, `${{ runner.os }}`, `${{ github.sha }}`, `${{ github.ref }}`, `${{ secrets.* }}`, `${{ env.* }}` — these are safe from GitHub expression injection when used in `run:` blocks.
|
|
66
75
|
|
|
67
|
-
###
|
|
76
|
+
### 4. Write Report
|
|
68
77
|
|
|
69
|
-
|
|
78
|
+
Replace the `IN_PROGRESS` body in this run's reserved `{outputFile}` with the final validation report. Include PASS/WARN/FAIL per section plus the original `validation_scope`, `run_timestamp`, and `validated_artifacts` metadata. Record every selected artifact using its exact project-relative path.
|
|
70
79
|
|
|
71
80
|
## 🚨 SYSTEM SUCCESS/FAILURE METRICS:
|
|
72
81
|
|
|
@@ -74,6 +83,7 @@ Write a validation report to `{outputFile}` with PASS/WARN/FAIL per section.
|
|
|
74
83
|
|
|
75
84
|
- Validation report written
|
|
76
85
|
- All checklist items evaluated
|
|
86
|
+
- All selected artifacts recorded in the report
|
|
77
87
|
|
|
78
88
|
### ❌ SYSTEM FAILURE:
|
|
79
89
|
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: 'step-01-validate'
|
|
3
3
|
description: 'Validate workflow outputs against checklist'
|
|
4
|
-
outputFile: '{test_artifacts}/framework-validation-report.md'
|
|
4
|
+
outputFile: '{test_artifacts}/framework-validation-report-{validation_scope}-{run_timestamp}.md'
|
|
5
5
|
validationChecklist: '{skill-root}/checklist.md'
|
|
6
6
|
---
|
|
7
7
|
|
|
@@ -26,6 +26,7 @@ Validate outputs using the workflow checklist and record findings.
|
|
|
26
26
|
|
|
27
27
|
- 🎯 Validate against `{validationChecklist}`
|
|
28
28
|
- 🚫 Do not skip checks
|
|
29
|
+
- 🚫 Never overwrite an existing validation report
|
|
29
30
|
|
|
30
31
|
## EXECUTION PROTOCOLS:
|
|
31
32
|
|
|
@@ -34,7 +35,7 @@ Validate outputs using the workflow checklist and record findings.
|
|
|
34
35
|
|
|
35
36
|
## CONTEXT BOUNDARIES:
|
|
36
37
|
|
|
37
|
-
- Available context: workflow outputs and checklist
|
|
38
|
+
- Available context: user-selected workflow outputs and checklist
|
|
38
39
|
- Focus: validation only
|
|
39
40
|
- Limits: do not modify outputs in this step
|
|
40
41
|
|
|
@@ -42,17 +43,25 @@ Validate outputs using the workflow checklist and record findings.
|
|
|
42
43
|
|
|
43
44
|
**CRITICAL:** Follow this sequence exactly.
|
|
44
45
|
|
|
45
|
-
### 1.
|
|
46
|
+
### 1. Select Scope and Resolve Report Path
|
|
47
|
+
|
|
48
|
+
Use the artifact paths the user supplied with the Validate request. If none were supplied, list the likely outputs for this workflow and ask which exact file or files to validate. When several candidates exist, do not guess.
|
|
49
|
+
|
|
50
|
+
Read the selected artifacts. Derive `validation_scope` from their shared story, epic, system, pull request, or other meaningful scope. Use an artifact basename without its extension when no broader scope is available. Normalize the value to lowercase ASCII with only letters, numbers, and single hyphens. Remove leading and trailing hyphens. Ask for a short scope label if normalization leaves an empty value.
|
|
51
|
+
|
|
52
|
+
Set `run_timestamp` to the current UTC time with milliseconds in `YYYYMMDDTHHmmssSSSZ` format and resolve `{outputFile}` with both values. Atomically reserve that path using an exclusive-create operation that fails if the file already exists. A separate existence check followed by a normal write is forbidden. On collision, generate a fresh timestamp, resolve a new path, and retry exclusive creation until it succeeds. Initialize the reserved file with `validation_scope`, `run_timestamp`, `validated_artifacts`, and `status: IN_PROGRESS`. This run may update only the file it reserved. If the workflow stops, leave that reservation in place. Never delete, truncate, or reuse a report from another run. Always refuse to overwrite prior validation history.
|
|
53
|
+
|
|
54
|
+
### 2. Load Checklist
|
|
46
55
|
|
|
47
56
|
Read `{validationChecklist}` and list all criteria.
|
|
48
57
|
|
|
49
|
-
###
|
|
58
|
+
### 3. Validate Outputs
|
|
50
59
|
|
|
51
60
|
Evaluate outputs against each checklist item.
|
|
52
61
|
|
|
53
|
-
###
|
|
62
|
+
### 4. Write Report
|
|
54
63
|
|
|
55
|
-
|
|
64
|
+
Replace the `IN_PROGRESS` body in this run's reserved `{outputFile}` with the final validation report. Include PASS/WARN/FAIL per section plus the original `validation_scope`, `run_timestamp`, and `validated_artifacts` metadata. Record every selected artifact using its exact project-relative path.
|
|
56
65
|
|
|
57
66
|
## 🚨 SYSTEM SUCCESS/FAILURE METRICS:
|
|
58
67
|
|
|
@@ -60,6 +69,7 @@ Write a validation report to `{outputFile}` with PASS/WARN/FAIL per section.
|
|
|
60
69
|
|
|
61
70
|
- Validation report written
|
|
62
71
|
- All checklist items evaluated
|
|
72
|
+
- All selected artifacts recorded in the report
|
|
63
73
|
|
|
64
74
|
### ❌ SYSTEM FAILURE:
|
|
65
75
|
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: 'step-01-validate'
|
|
3
3
|
description: 'Validate workflow outputs against checklist'
|
|
4
|
-
outputFile: '{test_artifacts}/nfr-assess-validation-report.md'
|
|
4
|
+
outputFile: '{test_artifacts}/nfr-assess-validation-report-{validation_scope}-{run_timestamp}.md'
|
|
5
5
|
validationChecklist: '{skill-root}/checklist.md'
|
|
6
6
|
---
|
|
7
7
|
|
|
@@ -26,6 +26,7 @@ Validate outputs using the workflow checklist and record findings.
|
|
|
26
26
|
|
|
27
27
|
- 🎯 Validate against `{validationChecklist}`
|
|
28
28
|
- 🚫 Do not skip checks
|
|
29
|
+
- 🚫 Never overwrite an existing validation report
|
|
29
30
|
|
|
30
31
|
## EXECUTION PROTOCOLS:
|
|
31
32
|
|
|
@@ -34,7 +35,7 @@ Validate outputs using the workflow checklist and record findings.
|
|
|
34
35
|
|
|
35
36
|
## CONTEXT BOUNDARIES:
|
|
36
37
|
|
|
37
|
-
- Available context: workflow outputs and checklist
|
|
38
|
+
- Available context: user-selected workflow outputs and checklist
|
|
38
39
|
- Focus: validation only
|
|
39
40
|
- Limits: do not modify outputs in this step
|
|
40
41
|
|
|
@@ -42,17 +43,25 @@ Validate outputs using the workflow checklist and record findings.
|
|
|
42
43
|
|
|
43
44
|
**CRITICAL:** Follow this sequence exactly.
|
|
44
45
|
|
|
45
|
-
### 1.
|
|
46
|
+
### 1. Select Scope and Resolve Report Path
|
|
47
|
+
|
|
48
|
+
Use the artifact paths the user supplied with the Validate request. If none were supplied, list the likely outputs for this workflow and ask which exact file or files to validate. When several candidates exist, do not guess.
|
|
49
|
+
|
|
50
|
+
Read the selected artifacts. Derive `validation_scope` from their shared story, epic, system, pull request, or other meaningful scope. Use an artifact basename without its extension when no broader scope is available. Normalize the value to lowercase ASCII with only letters, numbers, and single hyphens. Remove leading and trailing hyphens. Ask for a short scope label if normalization leaves an empty value.
|
|
51
|
+
|
|
52
|
+
Set `run_timestamp` to the current UTC time with milliseconds in `YYYYMMDDTHHmmssSSSZ` format and resolve `{outputFile}` with both values. Atomically reserve that path using an exclusive-create operation that fails if the file already exists. A separate existence check followed by a normal write is forbidden. On collision, generate a fresh timestamp, resolve a new path, and retry exclusive creation until it succeeds. Initialize the reserved file with `validation_scope`, `run_timestamp`, `validated_artifacts`, and `status: IN_PROGRESS`. This run may update only the file it reserved. If the workflow stops, leave that reservation in place. Never delete, truncate, or reuse a report from another run. Always refuse to overwrite prior validation history.
|
|
53
|
+
|
|
54
|
+
### 2. Load Checklist
|
|
46
55
|
|
|
47
56
|
Read `{validationChecklist}` and list all criteria.
|
|
48
57
|
|
|
49
|
-
###
|
|
58
|
+
### 3. Validate Outputs
|
|
50
59
|
|
|
51
60
|
Evaluate outputs against each checklist item.
|
|
52
61
|
|
|
53
|
-
###
|
|
62
|
+
### 4. Write Report
|
|
54
63
|
|
|
55
|
-
|
|
64
|
+
Replace the `IN_PROGRESS` body in this run's reserved `{outputFile}` with the final validation report. Include PASS/WARN/FAIL per section plus the original `validation_scope`, `run_timestamp`, and `validated_artifacts` metadata. Record every selected artifact using its exact project-relative path.
|
|
56
65
|
|
|
57
66
|
## 🚨 SYSTEM SUCCESS/FAILURE METRICS:
|
|
58
67
|
|
|
@@ -60,6 +69,7 @@ Write a validation report to `{outputFile}` with PASS/WARN/FAIL per section.
|
|
|
60
69
|
|
|
61
70
|
- Validation report written
|
|
62
71
|
- All checklist items evaluated
|
|
72
|
+
- All selected artifacts recorded in the report
|
|
63
73
|
|
|
64
74
|
### ❌ SYSTEM FAILURE:
|
|
65
75
|
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: 'step-01-validate'
|
|
3
3
|
description: 'Validate workflow outputs against checklist'
|
|
4
|
-
outputFile: '{test_artifacts}/test-design-validation-report.md'
|
|
4
|
+
outputFile: '{test_artifacts}/test-design-validation-report-{validation_scope}-{run_timestamp}.md'
|
|
5
5
|
validationChecklist: '{skill-root}/checklist.md'
|
|
6
6
|
---
|
|
7
7
|
|
|
@@ -26,6 +26,7 @@ Validate outputs using the workflow checklist and record findings.
|
|
|
26
26
|
|
|
27
27
|
- 🎯 Validate against `{validationChecklist}`
|
|
28
28
|
- 🚫 Do not skip checks
|
|
29
|
+
- 🚫 Never overwrite an existing validation report
|
|
29
30
|
|
|
30
31
|
## EXECUTION PROTOCOLS:
|
|
31
32
|
|
|
@@ -34,7 +35,7 @@ Validate outputs using the workflow checklist and record findings.
|
|
|
34
35
|
|
|
35
36
|
## CONTEXT BOUNDARIES:
|
|
36
37
|
|
|
37
|
-
- Available context: workflow outputs and checklist
|
|
38
|
+
- Available context: user-selected workflow outputs and checklist
|
|
38
39
|
- Focus: validation only
|
|
39
40
|
- Limits: do not modify outputs in this step
|
|
40
41
|
|
|
@@ -42,17 +43,25 @@ Validate outputs using the workflow checklist and record findings.
|
|
|
42
43
|
|
|
43
44
|
**CRITICAL:** Follow this sequence exactly.
|
|
44
45
|
|
|
45
|
-
### 1.
|
|
46
|
+
### 1. Select Scope and Resolve Report Path
|
|
47
|
+
|
|
48
|
+
Use the artifact paths the user supplied with the Validate request. If none were supplied, list the likely outputs for this workflow and ask which exact file or files to validate. When several candidates exist, do not guess.
|
|
49
|
+
|
|
50
|
+
Read the selected artifacts. Derive `validation_scope` from their shared story, epic, system, pull request, or other meaningful scope. Use an artifact basename without its extension when no broader scope is available. Normalize the value to lowercase ASCII with only letters, numbers, and single hyphens. Remove leading and trailing hyphens. Ask for a short scope label if normalization leaves an empty value.
|
|
51
|
+
|
|
52
|
+
Set `run_timestamp` to the current UTC time with milliseconds in `YYYYMMDDTHHmmssSSSZ` format and resolve `{outputFile}` with both values. Atomically reserve that path using an exclusive-create operation that fails if the file already exists. A separate existence check followed by a normal write is forbidden. On collision, generate a fresh timestamp, resolve a new path, and retry exclusive creation until it succeeds. Initialize the reserved file with `validation_scope`, `run_timestamp`, `validated_artifacts`, and `status: IN_PROGRESS`. This run may update only the file it reserved. If the workflow stops, leave that reservation in place. Never delete, truncate, or reuse a report from another run. Always refuse to overwrite prior validation history.
|
|
53
|
+
|
|
54
|
+
### 2. Load Checklist
|
|
46
55
|
|
|
47
56
|
Read `{validationChecklist}` and list all criteria.
|
|
48
57
|
|
|
49
|
-
###
|
|
58
|
+
### 3. Validate Outputs
|
|
50
59
|
|
|
51
60
|
Evaluate outputs against each checklist item.
|
|
52
61
|
|
|
53
|
-
###
|
|
62
|
+
### 4. Write Report
|
|
54
63
|
|
|
55
|
-
|
|
64
|
+
Replace the `IN_PROGRESS` body in this run's reserved `{outputFile}` with the final validation report. Include PASS/WARN/FAIL per section plus the original `validation_scope`, `run_timestamp`, and `validated_artifacts` metadata. Record every selected artifact using its exact project-relative path.
|
|
56
65
|
|
|
57
66
|
## 🚨 SYSTEM SUCCESS/FAILURE METRICS:
|
|
58
67
|
|
|
@@ -60,6 +69,7 @@ Write a validation report to `{outputFile}` with PASS/WARN/FAIL per section.
|
|
|
60
69
|
|
|
61
70
|
- Validation report written
|
|
62
71
|
- All checklist items evaluated
|
|
72
|
+
- All selected artifacts recorded in the report
|
|
63
73
|
|
|
64
74
|
### ❌ SYSTEM FAILURE:
|
|
65
75
|
|