universal-dev-standards 5.5.0 → 5.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/bundled/ai/options/testing/integration-testing.ai.yaml +2 -2
- package/bundled/ai/options/testing/unit-testing.ai.yaml +2 -2
- package/bundled/ai/standards/browser-compatibility-standards.ai.yaml +63 -0
- package/bundled/ai/standards/contract-testing-standards.ai.yaml +62 -0
- package/bundled/ai/standards/cross-flow-regression.ai.yaml +61 -0
- package/bundled/ai/standards/full-coverage-testing.ai.yaml +192 -0
- package/bundled/ai/standards/release-readiness-gate.ai.yaml +77 -0
- package/bundled/ai/standards/testing.ai.yaml +20 -13
- package/bundled/core/accessibility-standards.md +58 -0
- package/bundled/core/branch-completion.md +4 -0
- package/bundled/core/browser-compatibility-standards.md +220 -0
- package/bundled/core/checkin-standards.md +1 -0
- package/bundled/core/contract-testing-standards.md +182 -0
- package/bundled/core/cross-flow-regression.md +190 -0
- package/bundled/core/flow-based-testing.md +135 -2
- package/bundled/core/full-coverage-testing.md +183 -0
- package/bundled/core/performance-standards.md +65 -0
- package/bundled/core/release-quality-manifest.md +56 -10
- package/bundled/core/release-readiness-gate.md +184 -0
- package/bundled/locales/zh-CN/CHANGELOG.md +3 -3
- package/bundled/locales/zh-CN/README.md +1 -1
- package/bundled/locales/zh-TW/CHANGELOG.md +3 -3
- package/bundled/locales/zh-TW/README.md +1 -1
- package/bundled/locales/zh-TW/core/browser-compatibility-standards.md +11 -0
- package/bundled/locales/zh-TW/core/contract-testing-standards.md +11 -0
- package/bundled/locales/zh-TW/core/cross-flow-regression.md +11 -0
- package/bundled/locales/zh-TW/core/release-readiness-gate.md +11 -0
- package/package.json +6 -6
- package/src/commands/check.js +43 -0
- package/src/commands/init.js +2 -1
- package/src/commands/update.js +10 -0
- package/standards-registry.json +60 -4
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
# Flow-Based Testing
|
|
2
2
|
|
|
3
|
-
**Version**: 1.
|
|
4
|
-
**Last Updated**: 2026-05-
|
|
3
|
+
**Version**: 1.3.0
|
|
4
|
+
**Last Updated**: 2026-05-05
|
|
5
5
|
**Applicability**: All software projects with multi-step workflows
|
|
6
6
|
**Scope**: universal
|
|
7
7
|
**Industry Standards**: ISO/IEC/IEEE 29119-4 (Test Techniques), ISTQB Foundation Syllabus
|
|
@@ -118,6 +118,139 @@ describe("Flow Branch: Quota exceeded path", () => {
|
|
|
118
118
|
|
|
119
119
|
---
|
|
120
120
|
|
|
121
|
+
## Multi-Gate Flow Verification Model
|
|
122
|
+
|
|
123
|
+
Flow coverage is not a single pre-release check — it is a **progressive verification chain** across the entire SDLC. There are two fundamentally different questions that must be answered at different stages:
|
|
124
|
+
|
|
125
|
+
| Verification Type | Question | Executor | Timing |
|
|
126
|
+
|------------------|----------|----------|--------|
|
|
127
|
+
| **Coverage** | Are all terminal states tested? | Automated CI | Dev → Staging → Pre-UAT |
|
|
128
|
+
| **Correctness** | Are the terminal state definitions right? | Human UAT | UAT phase |
|
|
129
|
+
|
|
130
|
+
Confusing the two wastes UAT cycles on technical coverage issues that CI should have caught.
|
|
131
|
+
|
|
132
|
+
### Gate 0 — PRD Sign-off (Before Implementation Starts)
|
|
133
|
+
|
|
134
|
+
The three testability elements MUST be written into the PRD before a single line of code is written. Use `templates/requirement-template.md` §2.4 and §9.4:
|
|
135
|
+
|
|
136
|
+
| Element | PRD Section | When Required |
|
|
137
|
+
|---------|-------------|---------------|
|
|
138
|
+
| Preconditions + Ordered Steps | §2.4 | Flows with ≥ 3 steps |
|
|
139
|
+
| Decision Points list | §2.4 | Every branch condition |
|
|
140
|
+
| Terminal States list | §2.4 | All distinct end states |
|
|
141
|
+
| Decision Table (Each-Choice) | §9.4 | All flows |
|
|
142
|
+
| Upgrade to All-Combinations | §9.4 | Auth / payment / security |
|
|
143
|
+
| UAT acceptance script (pre-filled) | §9.4 | Before PRD approval |
|
|
144
|
+
|
|
145
|
+
> **Why at PRD stage?** Test engineers cannot derive branch coverage from a spec that only describes the happy path. Discovering missing decision points during test design wastes a full sprint.
|
|
146
|
+
|
|
147
|
+
### Gate 1 — PR Merge (Per Feature Branch)
|
|
148
|
+
|
|
149
|
+
Every PR that touches a flow with ≥ 3 steps MUST include automated tests covering the terminal states introduced or modified by that PR. Reviewers block merge if terminal states are added to §2.4 without corresponding tests.
|
|
150
|
+
|
|
151
|
+
### Gate 3 — Pre-UAT Deployment (Automated + QA Lead Sign-off)
|
|
152
|
+
|
|
153
|
+
CI must prove coverage completeness **before** UAT begins. UAT is for correctness validation, not technical testing.
|
|
154
|
+
|
|
155
|
+
Required CI checks:
|
|
156
|
+
- All Decision Table scenarios have a passing automated test
|
|
157
|
+
- Zero terminal states without test coverage
|
|
158
|
+
- Branch coverage ≥ 90% (or project-defined threshold)
|
|
159
|
+
- All-Combinations fully passing for auth / payment / security flows
|
|
160
|
+
|
|
161
|
+
> Deploying to UAT without Gate 3 forces business stakeholders to act as technical QA — a costly and demoralizing misuse of UAT time.
|
|
162
|
+
|
|
163
|
+
### Gate 4 — UAT Sign-off (Business Correctness, Pre-Production)
|
|
164
|
+
|
|
165
|
+
UAT validates that terminal state **definitions are correct** against real business rules, not that they are covered. Use the UAT Acceptance Script in §9.4 (derived directly from the Decision Table — no separate script creation needed):
|
|
166
|
+
|
|
167
|
+
- Business stakeholders sign off each row (terminal state)
|
|
168
|
+
- If UAT reveals a previously undefined terminal state: add it to §2.4 + Decision Table + automated test, re-run Gate 3, then resume UAT
|
|
169
|
+
- No new terminal states discovered during UAT = strong signal that §2.4 was thorough
|
|
170
|
+
|
|
171
|
+
### Gate Model Summary
|
|
172
|
+
|
|
173
|
+
```
|
|
174
|
+
PRD Sign-off
|
|
175
|
+
│ Gate 0: §2.4 + §9.4 complete (Decision Points, Terminal States,
|
|
176
|
+
│ Decision Table, UAT script pre-filled)
|
|
177
|
+
▼
|
|
178
|
+
Implementation + PR Reviews
|
|
179
|
+
│ Gate 1: Each PR covering a flow includes terminal state tests
|
|
180
|
+
▼
|
|
181
|
+
Staging / Integration
|
|
182
|
+
│ (no formal gate — CI green is sufficient)
|
|
183
|
+
▼
|
|
184
|
+
Pre-UAT Deployment
|
|
185
|
+
│ Gate 3: CI proves 100% terminal state coverage + branch coverage ≥ 90%
|
|
186
|
+
▼
|
|
187
|
+
UAT Execution
|
|
188
|
+
│ Gate 4: Business sign-off on terminal state correctness
|
|
189
|
+
│ New terminal states → back to Gate 3 before proceeding
|
|
190
|
+
▼
|
|
191
|
+
Production
|
|
192
|
+
```
|
|
193
|
+
|
|
194
|
+
---
|
|
195
|
+
|
|
196
|
+
## RQM Integration
|
|
197
|
+
|
|
198
|
+
Gate 3 (Pre-UAT CI coverage gate) MUST produce a **`flow_gate_report.json`** artifact consumed by the Release Quality Manifest (`release-quality-manifest.md`, field `flow_gate_report`).
|
|
199
|
+
|
|
200
|
+
### flow_gate_report.json Schema
|
|
201
|
+
|
|
202
|
+
```json
|
|
203
|
+
{
|
|
204
|
+
"generated_at": "2026-05-05T04:00:00Z",
|
|
205
|
+
"commit": "abc1234",
|
|
206
|
+
"flows": [
|
|
207
|
+
{
|
|
208
|
+
"flow_id": "login-authentication",
|
|
209
|
+
"spec_ref": "docs/specs/SPEC-001.md#2.4",
|
|
210
|
+
"decision_points": 3,
|
|
211
|
+
"terminal_states": 7,
|
|
212
|
+
"gate_0_complete": true,
|
|
213
|
+
"gate_1_pr_coverage": true,
|
|
214
|
+
"gate_3": {
|
|
215
|
+
"all_scenarios_green": true,
|
|
216
|
+
"terminal_states_covered": 7,
|
|
217
|
+
"terminal_states_defined": 7,
|
|
218
|
+
"branch_coverage_pct": 94,
|
|
219
|
+
"coverage_target": 90,
|
|
220
|
+
"all_combinations_required": false,
|
|
221
|
+
"status": "pass"
|
|
222
|
+
},
|
|
223
|
+
"gate_4_uat_signoff": true
|
|
224
|
+
}
|
|
225
|
+
],
|
|
226
|
+
"summary": {
|
|
227
|
+
"total_flows": 5,
|
|
228
|
+
"gate_0_complete": true,
|
|
229
|
+
"gate_1_pr_coverage": true,
|
|
230
|
+
"gate_3_ci_pass": true,
|
|
231
|
+
"gate_4_uat_signoff": true,
|
|
232
|
+
"status": "pass"
|
|
233
|
+
}
|
|
234
|
+
}
|
|
235
|
+
```
|
|
236
|
+
|
|
237
|
+
### Generation Script Hook
|
|
238
|
+
|
|
239
|
+
Add to CI after test run (Gate 3):
|
|
240
|
+
|
|
241
|
+
```bash
|
|
242
|
+
# scripts/generate-flow-gate-report.sh
|
|
243
|
+
node scripts/generate-flow-gate-report.mjs \
|
|
244
|
+
--coverage-report coverage/coverage-summary.json \
|
|
245
|
+
--flow-specs "docs/specs/**/*.md" \
|
|
246
|
+
--uat-signoffs ".release-readiness/*.md" \
|
|
247
|
+
--output flow_gate_report.json
|
|
248
|
+
```
|
|
249
|
+
|
|
250
|
+
The `summary.status` field feeds into `release-quality-manifest.yaml` under `flow_gate_report.status`.
|
|
251
|
+
|
|
252
|
+
---
|
|
253
|
+
|
|
121
254
|
## Quick Reference Checklist
|
|
122
255
|
|
|
123
256
|
```
|
|
@@ -0,0 +1,183 @@
|
|
|
1
|
+
# Full Coverage Testing Standards
|
|
2
|
+
|
|
3
|
+
> **AI-optimized version**: `ai/standards/full-coverage-testing.ai.yaml`
|
|
4
|
+
> **XSPEC**: XSPEC-178
|
|
5
|
+
> **Replaces**: Pyramid threshold model (UT≥80%, IT≥70%, E2E happy-path-only)
|
|
6
|
+
|
|
7
|
+
## Overview
|
|
8
|
+
|
|
9
|
+
Full Coverage Testing is a behavior-completeness paradigm designed for the AI-era, where the cost of generating tests equals the cost of generating code. Traditional pyramid thresholds assumed tests were expensive to write — this assumption no longer holds.
|
|
10
|
+
|
|
11
|
+
**Core principle**: Every public function must be tested for all three behavioral paths. Coverage is measured by behavior completeness, not percentage floors. CI enforces a ratchet: coverage can only increase, never decrease.
|
|
12
|
+
|
|
13
|
+
---
|
|
14
|
+
|
|
15
|
+
## Behavior-Completeness Model
|
|
16
|
+
|
|
17
|
+
Instead of "80% line coverage", require:
|
|
18
|
+
|
|
19
|
+
| Path | Description | Example |
|
|
20
|
+
|------|-------------|---------|
|
|
21
|
+
| **Happy path** | Normal input produces correct output | `calculateDiscount(100, 0.1) → 90` |
|
|
22
|
+
| **Edge case** | Boundary values do not cause unexpected errors | `calculateDiscount(0, 1.0) → 0 without throwing` |
|
|
23
|
+
| **Error path** | Invalid input raises clear error or error state | `calculateDiscount(-1, 2.0) → throws ArgumentError` |
|
|
24
|
+
|
|
25
|
+
Every public function requires all three. This replaces the "80% of business logic" target with a qualitative, behavior-driven requirement.
|
|
26
|
+
|
|
27
|
+
---
|
|
28
|
+
|
|
29
|
+
## Ratchet CI Policy
|
|
30
|
+
|
|
31
|
+
- The current coverage baseline is the minimum acceptable coverage
|
|
32
|
+
- Any PR that decreases coverage is blocked from merging
|
|
33
|
+
- Improvements update the baseline automatically on merge
|
|
34
|
+
- No fixed percentage floor — the coverage achieved today is tomorrow's floor
|
|
35
|
+
|
|
36
|
+
```bash
|
|
37
|
+
# Stored in .coverage-baseline.json
|
|
38
|
+
{ "line": 91.3, "branch": 88.7, "timestamp": "2026-05-06" }
|
|
39
|
+
|
|
40
|
+
# PR regression → blocked
|
|
41
|
+
Coverage regression: 91.3% → 89.1%. Ratchet threshold violated.
|
|
42
|
+
|
|
43
|
+
# PR improvement → baseline updated
|
|
44
|
+
Coverage improved: 91.3% → 92.0%. New baseline set.
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
---
|
|
48
|
+
|
|
49
|
+
## Anti-Fake Test Rules
|
|
50
|
+
|
|
51
|
+
### Forbidden: Tautology Assertions
|
|
52
|
+
|
|
53
|
+
Assertions that always pass regardless of behavior provide false coverage.
|
|
54
|
+
|
|
55
|
+
```typescript
|
|
56
|
+
// ❌ FORBIDDEN — always passes, tests nothing
|
|
57
|
+
expect(true).toBe(true)
|
|
58
|
+
expect(result).toBeDefined() // without specific value
|
|
59
|
+
|
|
60
|
+
// ✅ REQUIRED — verifies actual behavior
|
|
61
|
+
expect(result).toBe(90)
|
|
62
|
+
expect(result).toEqual({ discount: 10, total: 90 })
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
### Forbidden: Mocking Core Business Logic
|
|
66
|
+
|
|
67
|
+
Mocking your own code means the business logic is never actually executed.
|
|
68
|
+
|
|
69
|
+
```typescript
|
|
70
|
+
// ❌ FORBIDDEN — business logic never runs
|
|
71
|
+
jest.mock('./orderService', () => ({ calculateTotal: jest.fn(() => 100) }))
|
|
72
|
+
|
|
73
|
+
// ✅ ALLOWED — mock only external dependencies
|
|
74
|
+
// MOCK: External Stripe API — no sandbox available in CI
|
|
75
|
+
jest.mock('./payment-gateway', () => ({ charge: jest.fn().mockResolvedValue({ id: 'ch_test' }) }))
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
### Required: Mock Reason Comments
|
|
79
|
+
|
|
80
|
+
Every mock must explain why the dependency cannot be real.
|
|
81
|
+
|
|
82
|
+
```typescript
|
|
83
|
+
// ❌ FORBIDDEN — no explanation
|
|
84
|
+
jest.mock('./payment-gateway')
|
|
85
|
+
|
|
86
|
+
// ✅ REQUIRED — explicit reason
|
|
87
|
+
// MOCK: External payment gateway — network dependency, no sandbox in CI
|
|
88
|
+
jest.mock('./payment-gateway', () => ({ ... }))
|
|
89
|
+
```
|
|
90
|
+
|
|
91
|
+
### Mock Boundary: What Can Be Mocked
|
|
92
|
+
|
|
93
|
+
| ✅ Allowed to Mock | ❌ Forbidden to Mock |
|
|
94
|
+
|-------------------|---------------------|
|
|
95
|
+
| External HTTP APIs (payment, OAuth) | Core business calculation functions |
|
|
96
|
+
| Hardware interfaces (sensors, GPIO) | Your own service layer methods |
|
|
97
|
+
| Third-party SDKs without test mode | Database queries (use in-memory SQLite) |
|
|
98
|
+
| Docker daemon | Your own utility functions |
|
|
99
|
+
|
|
100
|
+
---
|
|
101
|
+
|
|
102
|
+
## STUB Marker Protocol
|
|
103
|
+
|
|
104
|
+
All temporary/placeholder implementations MUST be marked with the standard STUB marker. This is enforced by pre-push hooks and deploy.sh.
|
|
105
|
+
|
|
106
|
+
### Marking a STUB
|
|
107
|
+
|
|
108
|
+
```typescript
|
|
109
|
+
// WARNING: STUB — Remove before UAT
|
|
110
|
+
async function validatePayment(card: Card): Promise<boolean> {
|
|
111
|
+
return true; // Always approve — replace with real Stripe call
|
|
112
|
+
}
|
|
113
|
+
```
|
|
114
|
+
|
|
115
|
+
### Exempting a Genuine Limitation
|
|
116
|
+
|
|
117
|
+
When a dependency truly cannot be tested (hardware, live API without sandbox):
|
|
118
|
+
|
|
119
|
+
```typescript
|
|
120
|
+
// COVERAGE_EXEMPT: Hardware temperature sensor — no simulation available in CI
|
|
121
|
+
async function readTemperature(): Promise<number> {
|
|
122
|
+
return hardwareSensor.read();
|
|
123
|
+
}
|
|
124
|
+
```
|
|
125
|
+
|
|
126
|
+
The exemption reason MUST be non-empty and specific.
|
|
127
|
+
|
|
128
|
+
### Deployment Gates
|
|
129
|
+
|
|
130
|
+
| Environment | STUB Present | Action |
|
|
131
|
+
|-------------|-------------|--------|
|
|
132
|
+
| Feature branch push | Yes | ⚠️ Warning (not blocked) |
|
|
133
|
+
| `main` branch push | Yes | ❌ Blocked |
|
|
134
|
+
| Staging deploy | Yes | ⚠️ Warning (not blocked) |
|
|
135
|
+
| UAT deploy | Yes | ❌ Blocked |
|
|
136
|
+
| Production deploy | Yes | ❌ Blocked (critical log) |
|
|
137
|
+
|
|
138
|
+
---
|
|
139
|
+
|
|
140
|
+
## AC Traceability
|
|
141
|
+
|
|
142
|
+
Link each test to its Acceptance Criteria using the `@ac` JSDoc tag:
|
|
143
|
+
|
|
144
|
+
```typescript
|
|
145
|
+
/**
|
|
146
|
+
* @ac AC-US03-2
|
|
147
|
+
*/
|
|
148
|
+
it('should block PR when coverage regresses below baseline', () => {
|
|
149
|
+
// test body
|
|
150
|
+
})
|
|
151
|
+
|
|
152
|
+
// If no AC maps to this test:
|
|
153
|
+
/**
|
|
154
|
+
* @ac UNTRACED
|
|
155
|
+
*/
|
|
156
|
+
it('helper utility returns correct format', () => { ... })
|
|
157
|
+
```
|
|
158
|
+
|
|
159
|
+
CI reports AC coverage rate. If more than 20% of ACs lack `@ac`-tagged tests, a warning is shown.
|
|
160
|
+
|
|
161
|
+
---
|
|
162
|
+
|
|
163
|
+
## Migration from Pyramid Model
|
|
164
|
+
|
|
165
|
+
If your project previously used pyramid thresholds:
|
|
166
|
+
|
|
167
|
+
1. **Delete** any hardcoded coverage thresholds from `jest.config.js` / `vitest.config.ts` (`coverageThreshold` option)
|
|
168
|
+
2. **Install** `.coverage-baseline.json` with current coverage as the starting ratchet
|
|
169
|
+
3. **Add** `scripts/check-coverage-ratchet.sh` to CI
|
|
170
|
+
4. **Add** `scripts/check-stubs.sh` to deploy.sh and pre-push hook
|
|
171
|
+
5. **Add** `scripts/check-anti-fake-tests.sh` to pre-commit or CI
|
|
172
|
+
|
|
173
|
+
The ratchet starts at your current coverage. From that point on, it can only increase.
|
|
174
|
+
|
|
175
|
+
---
|
|
176
|
+
|
|
177
|
+
## Related Standards
|
|
178
|
+
|
|
179
|
+
- `testing.ai.yaml` — Test structure, FIRST principles, AAA pattern (pyramid thresholds deprecated here)
|
|
180
|
+
- `unit-testing.ai.yaml` — Unit test scope and organization
|
|
181
|
+
- `integration-testing.ai.yaml` — Integration test patterns
|
|
182
|
+
- `deployment-standards.ai.yaml` — Deploy gate requirements
|
|
183
|
+
- XSPEC-178 — Full specification and implementation phases
|
|
@@ -323,6 +323,70 @@ Analogous to the SRE Error Budget concept, a Performance Budget defines the tole
|
|
|
323
323
|
|
|
324
324
|
---
|
|
325
325
|
|
|
326
|
+
## Per-Release Capacity Sign-off
|
|
327
|
+
|
|
328
|
+
This section defines the **capacity gate** that must be satisfied before production release (Dimension 10 in `release-readiness-gate.md`, Tier-3).
|
|
329
|
+
|
|
330
|
+
### Capacity Forecast
|
|
331
|
+
|
|
332
|
+
Before each release candidate, produce a capacity forecast based on:
|
|
333
|
+
|
|
334
|
+
1. **Baseline**: 90-day rolling average of peak TPS and resource utilization (CPU, memory, DB connections, storage growth rate)
|
|
335
|
+
2. **Release impact estimate**: expected traffic delta from new features (e.g., +15% TPS from new notification flow)
|
|
336
|
+
3. **Seasonal adjustment**: any known traffic spikes within the next 30 days (marketing campaigns, seasonal peaks)
|
|
337
|
+
|
|
338
|
+
### Headroom Thresholds
|
|
339
|
+
|
|
340
|
+
| Metric | Target (PASS) | Warn Band | Fail Threshold |
|
|
341
|
+
|--------|--------------|-----------|----------------|
|
|
342
|
+
| CPU headroom at projected peak | ≥ 30% | 20–30% | < 20% |
|
|
343
|
+
| Memory headroom | ≥ 25% | 15–25% | < 15% |
|
|
344
|
+
| DB connection pool headroom | ≥ 40% | 25–40% | < 25% |
|
|
345
|
+
| p99 latency vs baseline | ≤ +5% | +5% to +10% | > +10% regression |
|
|
346
|
+
| Error rate at peak load | < 0.1% | 0.1–0.5% | > 0.5% |
|
|
347
|
+
|
|
348
|
+
### Load Test Requirement
|
|
349
|
+
|
|
350
|
+
Run the load test scenario defined in the Performance Testing sections above (Soak + Spike test minimum) before finalizing the capacity sign-off:
|
|
351
|
+
|
|
352
|
+
```bash
|
|
353
|
+
# Example: k6 capacity verification run
|
|
354
|
+
k6 run --vus 500 --duration 20m scripts/perf/soak-test.js
|
|
355
|
+
# Pass criterion: headroom metrics above, p99 within budget
|
|
356
|
+
```
|
|
357
|
+
|
|
358
|
+
### Sign-off Evidence
|
|
359
|
+
|
|
360
|
+
The capacity gate requires **two named sign-offs** — both Engineering Lead and SRE Lead:
|
|
361
|
+
|
|
362
|
+
```markdown
|
|
363
|
+
## Capacity Sign-off — <version>
|
|
364
|
+
|
|
365
|
+
**Projection date**: YYYY-MM-DD
|
|
366
|
+
**Baseline period**: last 90 days
|
|
367
|
+
|
|
368
|
+
| Metric | Baseline peak | Projected peak | Headroom | Status |
|
|
369
|
+
|--------|-------------|---------------|----------|--------|
|
|
370
|
+
| CPU | [X]% | [Y]% | [Z]% | PASS/WARN/FAIL |
|
|
371
|
+
| Memory | [X]% | [Y]% | [Z]% | PASS/WARN/FAIL |
|
|
372
|
+
| DB pool | [X]% | [Y]% | [Z]% | PASS/WARN/FAIL |
|
|
373
|
+
| p99 latency | [X]ms | [Y]ms | [±Z]% | PASS/WARN/FAIL |
|
|
374
|
+
|
|
375
|
+
**Load test artifact**: [link to load test report]
|
|
376
|
+
|
|
377
|
+
**Eng Lead sign-off**: _______________ Date: __________
|
|
378
|
+
**SRE Lead sign-off**: _______________ Date: __________
|
|
379
|
+
```
|
|
380
|
+
|
|
381
|
+
### When Tier-3 Applies as N/A
|
|
382
|
+
|
|
383
|
+
The capacity sign-off is `N/A` (with documented rationale) when:
|
|
384
|
+
- Project has < 100 DAU and no significant traffic growth expected
|
|
385
|
+
- Internal tooling with fixed user count
|
|
386
|
+
- Static content / documentation site
|
|
387
|
+
|
|
388
|
+
---
|
|
389
|
+
|
|
326
390
|
## Related Standards
|
|
327
391
|
|
|
328
392
|
- [Testing Standards](testing-standards.md) - Performance testing integration
|
|
@@ -330,6 +394,7 @@ Analogous to the SRE Error Budget concept, a Performance Budget defines the tole
|
|
|
330
394
|
- [Logging Standards](logging-standards.md) - Performance logging
|
|
331
395
|
- [Code Review Checklist](code-review-checklist.md) - Performance review
|
|
332
396
|
- [Deployment Standards](deployment-standards.md) - Performance validation pre-deployment
|
|
397
|
+
- [Release Readiness Gate](release-readiness-gate.md) - Dimension 1 (load) and Dimension 10 (capacity)
|
|
333
398
|
|
|
334
399
|
---
|
|
335
400
|
|
|
@@ -16,11 +16,14 @@ A Release Quality Manifest makes quality evidence:
|
|
|
16
16
|
|
|
17
17
|
## Schema
|
|
18
18
|
|
|
19
|
+
The RQM now covers **16 quality dimensions** matching `release-readiness-gate.md`. Automated gates appear here; human-verified gates appear in the Release Readiness Sign-off document.
|
|
20
|
+
|
|
19
21
|
```yaml
|
|
20
22
|
release: vibeops-commercial-1.2.0
|
|
21
23
|
generated_at: "2026-05-05T04:00:00Z"
|
|
22
24
|
commit: "abc1234"
|
|
23
25
|
gates:
|
|
26
|
+
# ── Automated quality gates ──────────────────────────────
|
|
24
27
|
unit_coverage:
|
|
25
28
|
actual: "73%"
|
|
26
29
|
target: "80%"
|
|
@@ -57,7 +60,42 @@ gates:
|
|
|
57
60
|
actual: true
|
|
58
61
|
target: true
|
|
59
62
|
status: pass
|
|
60
|
-
|
|
63
|
+
# ── Extended dimensions (aligned with release-readiness-gate.md) ──
|
|
64
|
+
a11y_critical: # Dimension 3: axe-core critical violations
|
|
65
|
+
actual: 0
|
|
66
|
+
target: 0
|
|
67
|
+
status: pass
|
|
68
|
+
a11y_serious: # Dimension 3: axe-core serious violations
|
|
69
|
+
actual: 0
|
|
70
|
+
target: 0
|
|
71
|
+
status: pass
|
|
72
|
+
contract_drift: # Dimension 4: consumer contracts failing (n/a if no consumers)
|
|
73
|
+
actual: 0
|
|
74
|
+
target: 0
|
|
75
|
+
status: pass # or "n/a" if no API consumers
|
|
76
|
+
cross_flow_cuj_pass_rate: # Dimension 6: critical user journey pass rate
|
|
77
|
+
actual: "100%"
|
|
78
|
+
target: "95%"
|
|
79
|
+
status: pass
|
|
80
|
+
browser_tier1_pass_rate: # Dimension 9: Tier-1 browser matrix (n/a for non-frontend)
|
|
81
|
+
actual: "100%"
|
|
82
|
+
target: "100%"
|
|
83
|
+
status: pass # or "n/a" for CLI/backend
|
|
84
|
+
capacity_headroom_cpu_pct: # Dimension 10: CPU headroom at projected peak (n/a for small projects)
|
|
85
|
+
actual: "42%"
|
|
86
|
+
target: "30%"
|
|
87
|
+
status: pass # or "n/a" for small-scale projects
|
|
88
|
+
smoke_pass_rate: # Dimension 14: post-deploy smoke (populated after staging deploy)
|
|
89
|
+
actual: "100%"
|
|
90
|
+
target: "100%"
|
|
91
|
+
status: pass
|
|
92
|
+
flow_gate_report: # Dimension 16: Multi-Gate Flow verification
|
|
93
|
+
gate_0_complete: true # all flows with ≥3 steps have §2.4 + §9.4 filled
|
|
94
|
+
gate_1_pr_coverage: true # all PRs touching flows include terminal-state tests
|
|
95
|
+
gate_3_ci_pass: true # Decision Table CI all green; branch coverage ≥ 90%
|
|
96
|
+
gate_4_uat_signoff: true # UAT sign-off table signed
|
|
97
|
+
status: pass
|
|
98
|
+
overall: WARN # worst gate status across all dimensions (2 warns, no fails)
|
|
61
99
|
```
|
|
62
100
|
|
|
63
101
|
## Status Semantics
|
|
@@ -68,15 +106,23 @@ overall: WARN # worst gate status (2 warns, no fails)
|
|
|
68
106
|
| `warn` | Within acceptable deviation (see per-gate policy) | Document reason; no release block |
|
|
69
107
|
| `fail` | Below hard minimum | **Blocks release** |
|
|
70
108
|
|
|
71
|
-
### Per-Gate Hard Minimums
|
|
72
|
-
|
|
73
|
-
| Gate | Warn Band | Fail Threshold |
|
|
74
|
-
|
|
75
|
-
| unit_coverage | target - 10pp to target | below target - 10pp |
|
|
76
|
-
| mutation_score | target - 5pp to target | below target - 5pp |
|
|
77
|
-
| sca_critical_cve | — | any critical CVE = fail |
|
|
78
|
-
| container_cve_critical | — | any critical CVE = fail |
|
|
79
|
-
| e2e_pass_rate | target - 3pp to target | below target - 3pp |
|
|
109
|
+
### Per-Gate Hard Minimums
|
|
110
|
+
|
|
111
|
+
| Gate | Warn Band | Fail Threshold | Release Readiness Dimension |
|
|
112
|
+
|------|-----------|----------------|----------------------------|
|
|
113
|
+
| unit_coverage | target - 10pp to target | below target - 10pp | (core RQM) |
|
|
114
|
+
| mutation_score | target - 5pp to target | below target - 5pp | (core RQM) |
|
|
115
|
+
| sca_critical_cve | — | any critical CVE = fail | Dim 2 (Security) |
|
|
116
|
+
| container_cve_critical | — | any critical CVE = fail | Dim 2 (Security) |
|
|
117
|
+
| e2e_pass_rate | target - 3pp to target | below target - 3pp | (core RQM) |
|
|
118
|
+
| a11y_critical | — | > 0 = fail | Dim 3 (a11y) |
|
|
119
|
+
| a11y_serious | project threshold | project threshold + 1-2 | Dim 3 (a11y) |
|
|
120
|
+
| contract_drift | — | any red consumer contract = fail (if n/a: skip) | Dim 4 (Contract) |
|
|
121
|
+
| cross_flow_cuj_pass_rate | 90–95% | < 90% | Dim 6 (Cross-flow Regression) |
|
|
122
|
+
| browser_tier1_pass_rate | — | < 100% (if n/a: skip) | Dim 9 (Browser Compat) |
|
|
123
|
+
| capacity_headroom_cpu_pct | 20–30% | < 20% (if n/a: skip) | Dim 10 (Capacity) |
|
|
124
|
+
| smoke_pass_rate | — | any smoke failure = fail | Dim 14 (Smoke) |
|
|
125
|
+
| flow_gate_report | gate_3_ci_pass=false | gate_0_complete=false OR gate_4_uat_signoff=false | Dim 16 (Multi-Gate Flow) |
|
|
80
126
|
|
|
81
127
|
## Automated Generation
|
|
82
128
|
|
|
@@ -0,0 +1,184 @@
|
|
|
1
|
+
# Release Readiness Gate
|
|
2
|
+
|
|
3
|
+
> **Language**: English | [繁體中文](../locales/zh-TW/core/release-readiness-gate.md)
|
|
4
|
+
|
|
5
|
+
**Version**: 1.0.0
|
|
6
|
+
**Last Updated**: 2026-05-05
|
|
7
|
+
**Applicability**: All software projects preparing a production release
|
|
8
|
+
**Scope**: universal
|
|
9
|
+
**Industry Standards**: ISO/IEC 25010 (Product Quality), ISTQB Advanced Test Manager
|
|
10
|
+
**References**: `core/release-quality-manifest.md`, `core/flow-based-testing.md`
|
|
11
|
+
|
|
12
|
+
---
|
|
13
|
+
|
|
14
|
+
## Purpose
|
|
15
|
+
|
|
16
|
+
This standard defines a **single, aggregated Release Readiness Gate** that unifies all quality dimensions into one explicit go/no-go decision before production deployment.
|
|
17
|
+
|
|
18
|
+
Without this gate, quality evidence is spread across 16+ separate standards. Teams pass individual checks but ship with unverified dimensions, because no one document says "you must pass *all of these* before release."
|
|
19
|
+
|
|
20
|
+
The Release Readiness Gate:
|
|
21
|
+
- **Aggregates** 16 quality dimensions into a tiered checklist
|
|
22
|
+
- **Connects** human sign-off (this document) to machine-readable evidence (`release-quality-manifest.md`)
|
|
23
|
+
- **Distinguishes** blocking criteria from advisory warnings
|
|
24
|
+
- **Scales** via Tier-1 / Tier-2 / Tier-3 classification to fit projects of different types and risk levels
|
|
25
|
+
|
|
26
|
+
---
|
|
27
|
+
|
|
28
|
+
## Relationship to Release Quality Manifest (RQM)
|
|
29
|
+
|
|
30
|
+
| Artifact | Format | Audience | Purpose |
|
|
31
|
+
|----------|--------|----------|---------|
|
|
32
|
+
| **Release Readiness Sign-off** (this document's template) | Markdown checklist | Humans (PM, QA, Eng Lead, Business) | Go/no-go decision, accountability, audit trail |
|
|
33
|
+
| **Release Quality Manifest** (`release-quality-manifest.md`) | YAML/JSON | CI, tooling, customers | Machine-readable aggregation, automated gate enforcement |
|
|
34
|
+
|
|
35
|
+
These two artifacts are generated **in parallel** for every release. The Sign-off covers human-verified dimensions; the RQM covers automated dimensions. Both must be `PASS` / `WARN` (never `FAIL`) before production deployment.
|
|
36
|
+
|
|
37
|
+
---
|
|
38
|
+
|
|
39
|
+
## Tier Classification
|
|
40
|
+
|
|
41
|
+
| Tier | Requirement | Miss = ? | Who Applies |
|
|
42
|
+
|------|-------------|---------|-------------|
|
|
43
|
+
| **Tier-1** | Must pass; release blocked if `FAIL` | Hard block | All projects |
|
|
44
|
+
| **Tier-2** | Should pass; `WARN` documented with rationale; no block | Documented WARN | All projects |
|
|
45
|
+
| **Tier-3** | Applicable when feature set or domain requires it; `N/A` is valid | N/A accepted | Depends on project type |
|
|
46
|
+
|
|
47
|
+
---
|
|
48
|
+
|
|
49
|
+
## 16-Dimension Release Readiness Matrix
|
|
50
|
+
|
|
51
|
+
| # | Dimension | Tier | Gate Type | Blocking Criterion | Evidence | Standard | Responsible |
|
|
52
|
+
|---|-----------|------|-----------|-------------------|----------|---------|-------------|
|
|
53
|
+
| 1 | **Performance / Load** | 2 | Automated | p95 latency regression > 10%; headroom < 20% | Load test report | `performance-standards.md` | Eng Lead + SRE |
|
|
54
|
+
| 2 | **Security** (SAST/DAST/SCA/secrets) | 1 | Automated | Any Critical/High CVE, SAST High unfixed, secret in diff | SARIF, Trivy, SBOM | `pipeline-security-gates.md` | SecEng / Eng Lead |
|
|
55
|
+
| 3 | **Accessibility (a11y)** | 2 | Automated + Manual | axe-core critical > 0; keyboard nav path broken | axe report, screen reader log | `accessibility-standards.md` §Release-Blocking Threshold | QA + UX |
|
|
56
|
+
| 4 | **API / Contract Testing** | 3 | Automated | Upstream consumer contract red; N-1 compat broken | Pact broker report | `contract-testing-standards.md` | API owner |
|
|
57
|
+
| 5 | **Database Migration** | 1 | Automated | up/rollback/idempotency test fails; data-preservation test fails | `data-migration-testing.md` gate results | `data-migration-testing.md` | DB Lead |
|
|
58
|
+
| 6 | **Cross-flow Regression** | 2 | Automated | Critical user journey pass rate < 95%; business-critical flow combo fails | Cross-flow regression report | `cross-flow-regression.md` | QA Lead |
|
|
59
|
+
| 7 | **Operational Readiness** | 1 | Manual | Runbook missing; alerting unconfigured; no rollback procedure | Runbook link, alert rule review | `runbook-standards.md`, `alerting-standards.md` | SRE / Ops |
|
|
60
|
+
| 8 | **Localization / i18n** | 2 | Automated | MISSING or MAJOR i18n gap in release (semver gap) | `check-translation-sync.sh` output | `translation-lifecycle-standards.md` | i18n Lead |
|
|
61
|
+
| 9 | **Browser / Device Compatibility** | 3 | Automated | Tier-1 browser/device pass rate < 100% | Playwright matrix report | `browser-compatibility-standards.md` | Frontend QA |
|
|
62
|
+
| 10 | **Capacity Sign-off** | 3 | Manual | Headroom < 30% at projected peak; no Eng+SRE sign-off | Capacity forecast + sign-off | `performance-standards.md` §Per-Release Capacity Sign-off | SRE + Eng Lead |
|
|
63
|
+
| 11 | **Compliance / Privacy** | 3 | Manual | GDPR/CCPA violation; audit log missing; retention policy broken | Privacy review checklist | `privacy-standards.md` | DPO / Legal |
|
|
64
|
+
| 12 | **Documentation Completeness** | 2 | Manual | CHANGELOG missing for release; customer-facing docs not updated | CHANGELOG diff, docs review | `changelog-standards.md`, `documentation-lifecycle.md` | Tech Writer / PM |
|
|
65
|
+
| 13 | **Rollback / Disaster Recovery** | 1 | Manual | No tested rollback procedure for this release; RTO > threshold | DR drill record; rollback script | `rollback-standards.md`, `disaster-recovery-drill.md` | SRE |
|
|
66
|
+
| 14 | **Production Smoke / Canary** | 1 | Automated | Post-deploy smoke fails; canary error rate > SLO | Smoke test results; canary dashboard | `smoke-test.md`, `cd-deployment-strategies.md` | SRE / DevOps |
|
|
67
|
+
| 15 | **Feature Flag Governance** | 2 | Manual | Default state not reviewed; kill-switch not tested | Flag audit checklist | `feature-flag-standards.md` | PM + Eng Lead |
|
|
68
|
+
| 16 | **Multi-Gate Flow Verification** | 2 | Automated + Manual | Gate 0 missing for any flow with ≥ 3 steps; Gate 3 CI fail; Gate 4 UAT sign-off missing | `flow_gate_report.json`; UAT sign-off table | `flow-based-testing.md` §Multi-Gate | QA Lead + Business |
|
|
69
|
+
|
|
70
|
+
> **Note on Tier-3**: Mark as `N/A` when not applicable (e.g., browser matrix for a CLI tool; contract testing for a standalone service with no API consumers). `N/A` requires a rationale comment in the sign-off.
|
|
71
|
+
|
|
72
|
+
---
|
|
73
|
+
|
|
74
|
+
## Release Readiness Sign-off Template
|
|
75
|
+
|
|
76
|
+
> Copy this template for each release. File as `.release-readiness/<version>.md` in the repo root, or attach to the release artifact.
|
|
77
|
+
|
|
78
|
+
```markdown
|
|
79
|
+
# Release Readiness Sign-off
|
|
80
|
+
|
|
81
|
+
**Release**: [tag/version]
|
|
82
|
+
**Date**: [YYYY-MM-DD]
|
|
83
|
+
**Environment**: Pre-Production → Production
|
|
84
|
+
**RQM Artifact**: [link or commit SHA]
|
|
85
|
+
|
|
86
|
+
## Tier-1 Gates (ALL must be PASS)
|
|
87
|
+
|
|
88
|
+
| # | Dimension | Status | Evidence | Sign-off |
|
|
89
|
+
|---|-----------|--------|----------|---------|
|
|
90
|
+
| 2 | Security (SAST/DAST/SCA) | PASS / FAIL | [link] | [name] |
|
|
91
|
+
| 5 | Database Migration | PASS / FAIL | [link] | [name] |
|
|
92
|
+
| 7 | Operational Readiness | PASS / FAIL | [link] | [name] |
|
|
93
|
+
| 13 | Rollback / DR | PASS / FAIL | [link] | [name] |
|
|
94
|
+
| 14 | Production Smoke/Canary | PASS / FAIL | [link] | [name] |
|
|
95
|
+
|
|
96
|
+
## Tier-2 Gates (WARN must have rationale)
|
|
97
|
+
|
|
98
|
+
| # | Dimension | Status | Evidence | Rationale (if WARN) | Sign-off |
|
|
99
|
+
|---|-----------|--------|----------|---------------------|---------|
|
|
100
|
+
| 1 | Performance / Load | PASS / WARN / FAIL | [link] | | [name] |
|
|
101
|
+
| 3 | Accessibility | PASS / WARN / FAIL | [link] | | [name] |
|
|
102
|
+
| 6 | Cross-flow Regression | PASS / WARN / FAIL | [link] | | [name] |
|
|
103
|
+
| 8 | Localization / i18n | PASS / WARN / FAIL | [link] | | [name] |
|
|
104
|
+
| 12 | Documentation | PASS / WARN / FAIL | [link] | | [name] |
|
|
105
|
+
| 15 | Feature Flag Governance | PASS / WARN / FAIL | [link] | | [name] |
|
|
106
|
+
| 16 | Multi-Gate Flow Verification | PASS / WARN / FAIL | [link] | | [name] |
|
|
107
|
+
|
|
108
|
+
## Tier-3 Gates (N/A with rationale allowed)
|
|
109
|
+
|
|
110
|
+
| # | Dimension | Status | Evidence | Rationale (if N/A) | Sign-off |
|
|
111
|
+
|---|-----------|--------|----------|---------------------|---------|
|
|
112
|
+
| 4 | API / Contract Testing | PASS / WARN / N/A | [link] | | [name] |
|
|
113
|
+
| 9 | Browser / Device Compat | PASS / WARN / N/A | [link] | | [name] |
|
|
114
|
+
| 10 | Capacity Sign-off | PASS / WARN / N/A | [link] | | [name] |
|
|
115
|
+
| 11 | Compliance / Privacy | PASS / WARN / N/A | [link] | | [name] |
|
|
116
|
+
|
|
117
|
+
## Overall Decision
|
|
118
|
+
|
|
119
|
+
- [ ] **GO** — All Tier-1 PASS; all WARN documented; all N/A have rationale
|
|
120
|
+
- [ ] **NO-GO** — One or more Tier-1 FAIL, or undocumented WARN
|
|
121
|
+
|
|
122
|
+
**Decision made by**: [name, role]
|
|
123
|
+
**Date**: [YYYY-MM-DD]
|
|
124
|
+
```
|
|
125
|
+
|
|
126
|
+
---
|
|
127
|
+
|
|
128
|
+
## Status Semantics
|
|
129
|
+
|
|
130
|
+
| Status | Meaning | Release Impact |
|
|
131
|
+
|--------|---------|----------------|
|
|
132
|
+
| `PASS` | Meets or exceeds all criteria | None |
|
|
133
|
+
| `WARN` | Below target but above hard minimum; rationale documented | Allowed; logged |
|
|
134
|
+
| `FAIL` | Below hard minimum; unresolved | **Blocks release** |
|
|
135
|
+
| `N/A` | Dimension not applicable to this project/release; rationale documented | Allowed |
|
|
136
|
+
|
|
137
|
+
---
|
|
138
|
+
|
|
139
|
+
## When to Create the Sign-off
|
|
140
|
+
|
|
141
|
+
| Milestone | Action |
|
|
142
|
+
|-----------|--------|
|
|
143
|
+
| Release candidate tagged | Create `.release-readiness/<version>.md` from template; fill evidence links |
|
|
144
|
+
| Pre-UAT deployment | Gate 3 CI results populated; Tier-1 automated gates verified |
|
|
145
|
+
| UAT sign-off (Gate 4) | Tier-3 manual gates completed; Multi-Gate Flow row finalized |
|
|
146
|
+
| Production deployment decision | Overall GO/NO-GO decision signed by release owner |
|
|
147
|
+
|
|
148
|
+
The sign-off is **not** an afterthought — Gate 0 (PRD completeness) and Gate 1 (PR-level tests) must be satisfied long before the sign-off document is created. The sign-off aggregates evidence that was being collected throughout the release cycle.
|
|
149
|
+
|
|
150
|
+
---
|
|
151
|
+
|
|
152
|
+
## Anti-Patterns
|
|
153
|
+
|
|
154
|
+
- **Creating the sign-off the day of deployment** — evidence should be collected incrementally throughout the release cycle
|
|
155
|
+
- **Marking WARN without rationale** — WARN without documented reason is functionally equivalent to ignoring the gate
|
|
156
|
+
- **Skipping Tier-3 entirely without N/A rationale** — if browser testing is omitted for a web app, that must be explicitly justified
|
|
157
|
+
- **Treating the Sign-off as a rubber stamp** — every row requires a named sign-off owner; anonymous collective ownership means no real accountability
|
|
158
|
+
- **Using a shared sign-off for multiple releases** — one sign-off per release tag; do not reuse across versions
|
|
159
|
+
|
|
160
|
+
---
|
|
161
|
+
|
|
162
|
+
## See Also
|
|
163
|
+
|
|
164
|
+
- `release-quality-manifest.md` — machine-readable RQM (the automated counterpart to this sign-off)
|
|
165
|
+
- `flow-based-testing.md` — Multi-Gate Flow Model (Dimension 16)
|
|
166
|
+
- `branch-completion.md` — branch-level gate (prerequisite; not equivalent to release readiness)
|
|
167
|
+
- `verification-evidence.md` — evidence standards (all evidence links must meet this standard)
|
|
168
|
+
- `deployment-standards.md` — post-deploy gate integration
|
|
169
|
+
|
|
170
|
+
---
|
|
171
|
+
|
|
172
|
+
## Version History
|
|
173
|
+
|
|
174
|
+
| Version | Date | Changes |
|
|
175
|
+
|---------|------|---------|
|
|
176
|
+
| 1.0.0 | 2026-05-05 | Initial release: 16-dimension matrix, tiered sign-off template, RQM integration |
|
|
177
|
+
|
|
178
|
+
---
|
|
179
|
+
|
|
180
|
+
## License
|
|
181
|
+
|
|
182
|
+
This standard is released under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/).
|
|
183
|
+
|
|
184
|
+
**Source**: [universal-dev-standards](https://github.com/AsiaOstrich/universal-dev-standards)
|